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Bio-Inspired  Sensing  and  Display  of  Polarization  Imagery 

Abstract 

Polarization  is  an  important  feature  of  electromagnetic  waves,  and  it  can  be  affected  by  surface  shapes, 
materials,  local  curvature  and  features,  and  relative  location  of  sources  and  objects,  and  thus  it  can  provide  useful 
information  about  the  observed  scene  and  objects.  Of  the  several  characteristics  of  visible  light,  only  two  -  intensity 
and  wavelength  -  are  encoded  by  the  human  eye  and  subsequently  mapped  by  the  visual  system  into  perceptual 
qualities.  Intensity  is  mapped  to  brightness  and  wavelength  information  is  mapped  to  perceived  color.  In  contrast, 
without  appropriate  instruments  human  eyes  cannot  effectively  utilize  light’s  polarization.  However,  it  is  well 
known  that  eyes  of  certain  animal  species  are  sensitive  to  the  polarization  of  light.  Polarization  sensing  has  mostly 
been  shown  to  provide  information  for  navigation,  as  originally  discovered  by  the  Nobel  laureate  Karl  von  Frisch  in 
his  work  with  honeybees.  In  addition,  it  has  been  hypothesized  that  some  species  may  have  evolved  polarization 
sensitivity  as  a  mechanism  of  enhancing  the  contrast  of  targets  in  media  that  scatter  light  (e.g.,  under  water).  Indeed, 
in  our  earlier  work,  we  have  shown  that  an  optical  imaging  system  utilizing  such  contrast  enhancement  through 
polarization  differencing  can  increase  the  distance  over  which  targets  can  be  detected,  and  their  critical  features 
discriminated.  Thus,  “polarization  imagery”  is  a  naturally  occurring  and  demonstrably  successful  strategy  for 
enhancing  vision. 

Since  the  human  eye  is  effectively  polarization-blind,  in  mapping  polarization  information  into  visual 
displays,  the  output  should  be  exhibited  in  a  way  compatible  with  the  biology  of  the  human  visual  system.  In  our 
group’s  earlier  work,  we  have  shown  one  such  bio-inspired  mapping,  in  which  polarization  information  was  pseudo 
color-coded,  based  on  the  opponent-colors  model  of  human  vision.  The  results  established  several  promising 
imaging  strategies  for  making  available  in  a  natural  way  the  contrast  enhancement  of  polarization  imagery  of  objects 
in  scattering  media. 

During  our  research  efforts  in  the  period  of  March  1,  2002  till  December  31,  2004,  we  have  explored  and 
investigated  various  bio-inspired  display  methodologies  for  mapping  polarization  information  into  visual 
information  that  can  be  readily  perceived  by  the  human  visual  system,  and  we  have  studied  those  mappings  most 
suitable  for  specific  applications  for  object  detection,  scene  classifications,  and  visibility  enhancement.  We  have 
introduced  and  developed  various  imaging  algorithms,  sensing  schemes  and  visualization  and  display  methodologies 
inspired  and  informecTby  biological  consideration.  Our  research  efforts  have  demonstrated  that  these  bio-inspired 
polarization  sensing  and  imaging  techniques  enable  us  to  achieve  better  target  detection,  enhanced  visibility  in 
otherwise  low-contrast  conditions,  longer  detection  range  in  optically  scattering  media,  man-made  polarization¬ 
sensing  adaptation  based  on  changing  environments,  surface  deformation/variation  detection  (e.g.,  detection  of 
finger  prints  on  a  smooth  surface  using  polarization-based  vision),  “shadow  removal”  by  displaying  polarization 
information  instead  of  conventional  intensity  information,  and  many  more  novel  outcomes.  These  results  have 
shown  the  numerous  possibilities  and  potential  applications  of  these  bio-inspired  methods  in  various  sensing, 
imaging,  and  display  technologies.  We  have  also  explored  several  ideas  for  mapping  polarization  information, 
including  using  pseudocolor  mapping,  static  and  dynamic  textures  with  varying  in  orientations,  flickering, 
modulating  luminance  and/or  color  contrast  of  scenes  in  terms  of  certain  aspects  of  polarization  values,  and  fusing 
polarization  information  into  optical  imagery.  Our  efforts  have  shown  that  the  polarization  can  bring  another 
“dimension  of  information”  into  the  domain  of  imaging  and  sensing  for  detection,  visibility  enhancement,  and 
display  methodologies.  These  techniques  have  the  potentials  to  bring  novel  advances  in  various  areas  of  Air  Force 
needs  such  as  imaging  systems  for  navigation  in  low-visibility  conditions. 
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Background  and  Introduction 

Polarization  is  an  important  feature  of  electromagnetic  waves,  and  can  be  affected  by 
various  factors  in  the  scene  such  as  surface  shapes,  surface  curvatures  and  materials,  and  position 
of  objects  with  respect  to  the  source(s)  and  the  observer  [1],  and  therefore  it  can  provide  useful 
information  about  the  observed  scene  and  objects  -  information  which  otherwise  would  be  lost. 
Two  characteristics  of  visible  light,  i.e.,  intensity  and  wavelength,  can  be  detected  and  encoded 
by  the  human  visual  system  and  interpreted  as  perceptual  qualities  of  brightness  and  color. 
Another  important  characteristic  of  light,  namely,  polarization,  however,  cannot  be  effectively 
exploited  by  unaided  human  eyes.  The  imaging  system  of  the  human  eye  is  effectively 
“polarization-blind”,  and  cannot  utilize  the  polarization  of  light  without  the  aid  of  an  artificial, 
polarization-sensitive  instrument.  The  eyes  of  some  animal  species  (e.g.,  honeybees,  desert  ants, 
backswimmer  flying  water  bug,  salmon,  cuttlefish),  however,  can  sense  the  polarization  of  light 
(see  e.g.,  [2-23]),  and  it  has  been  shown  by  biologists  and  zoologists  that  such  polarization  vision 
in  certain  species  may  facilitate  their  navigation  (see  e.g.,  [4-7]).  Moreover,  it  has  been  suggested 
that  some  species  may  utilize  the  polarization  sensitivity  to  enhance  the  contrast  of  targets  in 
scattering  media  (see  e.g.,  [18]). 

It  is  known  that  the  function  of  man-made  sensing,  imaging,  detection,  and  navigational 
systems  can  be  severely  limited  and  constrained  by  the  scattering  of  optical  waves  in  media 
surrounding  the  targets  [I],  which  leads  to  major  challenges  in  the  wave-based  direction  finding 
and  navigation,  imaging  and  target  detection,  and  reduction  of  contrast  in  such  media.  Any 
strategies  to  alleviate  such  contrast-degrading  effects  of  scattering,  and  thus  to  enhance  the 
imaging,  information  gathering,  and  detection  and  navigation  under  such  scattering  conditions  are 
clearly  welcome  additions  to  the  repertoire  of  imaging  and  sensing  techniques.  It  is  believed  that 
the  biology  of  visual  systems  of  some  animal  species  that  possess  polarization  vision  may  provide 
novel  solutions  to  such  challenging  problems  of  imaging  and  navigation  in  scattering  media.  As 
an  excellent  example,  the  underwater  environment  is  a  domain  where  scatterers  are  present,  and 
thus  the  performance  of  imaging  systems  is  limited.  Therefore,  the  aquatic  species  may  have 
adaptations  for  improving  the  quality  of  vision  in  scattering  media,  and  much  insight  and 
inspiration  can  be  gained  by  learning  from  what  biologists  and  zoologists  have  found  about  such 
polarization-sensitive  visual  systems. 
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The  biophysical  basis  of  polarization  detection  in  invertebrates  with  polarization  vision  is 
an  inherent  dichroism,  the  alignment  of  chromophores  (the  molecules  of  visual  pigments)  along 
the  photoreceptor  microvilli  (see  e.g.,  [3-8],  [14]).  Furthermore,  in  some  vertebrates,  for  which 
there  is  behavioral  evidence  for  polarization  sensitivity,  such  dichroism  is  not  present  since  the 
chromophores  are  free  to  rotate  in  the  plane  of  the  disc  membrane  of  the  outer  segment  of 
photoreceptors.  However,  many  teleosts  have  interesting  double-cone  mosaic  in  their  retinas  [35- 
36],  and  there  have  been  exciting  ideas  to  explain  the  functions  of  such  double-cone  mosaic  in  the 
context  of  polarization  sensitivity  [10,  21].  Moreover,  it  has  been  shown  by  Labhart  and  his 
colleagues  that  some  insects  with  polarization  vision  possess  polarization-opponent  intemeurons 
to  allow  the  two  polarization  components  of  received  signals  interact  antagonistically  leading  to 
enhanced  polarization  sensitivity  [6],  This  is  indeed  the  concept  of  polarization  opponency, 
analogous  to  color  opponency  in  human  visual  system,  and  in  fact  such  opponent  coding  is  the 
neurophysiological  equivalence  of  a  common-mode  rejection  and  differential  amplification  in 
circuit  domains.  The  opponent  coding  of  the  polarization  signal  may  yield  a  polarization 
difference  image  in  which  the  visibility  and  the  contrast  are  enhanced  significantly. 

Motivated  and  inspired  by  the  visual  systems  of  some  of  these  animal  species  with 
polarization  vision  and  particularly  by  the  concept  of  polarization  opponency,  we  have  been 
investigating  novel  imaging  methodologies  based  on  the  concept  of  polarization-difference 
signals  [24-34],  In  our  earlier  work,  we  have  shown  that  biologically  inspired  polarization- 
difference  imaging  (PDI)  can  extend  the  distance  over  which  objects  can  be  reliably  detected  in 
scattering  media  up  to  2-3  fold,  even  under  conditions  when  the  fractional  polarization  reaching 
the  detector  is  less  than  1%  [24-29],  Extending  the  distance  for  detection  and  classification  of 
objects  near  the  limit  of  visibility  increases  the  time  for  critical,  survival-relevant  decisions,  and  is 
thus  a  major  advantage  in  many  real-time  applications. 

In  our  efforts  during  the  period  March  1,  2002  till  December  31,  2004,  we  have 
developed  a  series  of  bio-inspired  algorithms  and  schemes  for  enhancing  sensing  and  imaging  of 
polarization  information  in  various  conditions  [30-34],  Here,  we  review  some  of  our  findings 
under  this  program.  Further  details  can  be  found  in  some  of  our  publications,  which  either  have 
appeared  in  print,  have  been  submitted  for  publication  or  are  in  preparation  for  submission  for 
publication.  These  can  be  found  in  the  appendix. 
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A  Sample  of  our  Findings  during  the  period  March  1,  2002  till 
December  31,  2004. 

During  this  period,  we  have  accomplished  a  variety  of  different  tasks  regarding  the  bio¬ 
inspired  sensing  and  mapping  of  optical  polarization  information.  These  investigations  are 
ongoing,  and  here  we  present  some  of  our  results  and  works  in  progress.  In  the  appendix,  more 
information  is  provided  in  our  publications. 

1. Tar  set  Detection,  Visibility  Enhancement,  and  Detection-Range  Increase  in  Scattering  Media 

We  have  continued  our  efforts  in  this  area  by  developing  and  improving  various  bio¬ 
inspired  algorithms  using  polarization  information  in  imaging  and  sensing  targets  in  scattering 
media.  These  include  improving  the  polarization-opponent  techniques  using  different  thresholds 
in  the  distribution  of  polarization-difference  (PD)  signals  in  an  image.  Fig.  1  shows  a  sample  of 
our  results  using  the  PD  technique  with  two  different  threshold  levels  in  the  histograms  of  the  PD 
signals  of  the  image.  Here  we  show  the  image  of  an  aluminum  disk,  specifically  made  for  our 
experiments.  This  disk  is  5.1 -cm  in  diameter  with  its  surface  sandblasted,  except  for  6  patch 
areas  each  with  1-cm2  square.  There  are  7  patches,  six  of  which  are  abraded  with  the  emory 
paper  in  various  specific  directions,  and  the  7th  patch  in  the  center  is  also  sandblasted.  The 
sandblasting  caused  the  surface  to  be  Lambertian.  The  patches  were  raised  a  few  mills.  The 
abrasions  (i.e.,  scratches)  on  these  six  patches  form  three  pairs.  Each  pair  has  two  orthogonal 
orientations  for  the  scratches.  Specifically,  the  orientations  of  the  scratches  on  these  six  patches 
are  0”,  30”,  60',  90",  120”,  and  150"  (with  respect  to  the  vertical  axis.),  where  the  patch  pairs 
with  orthogonal  oricntations,J.e.,  the  pairs  with  (0",90"j  p-.(30\120")  ,,and  (60°,  150°)  , are  located 

diametrically  on  the  disk.  The  polarization-sum  (PS)  and  the  polarization-difference  (PD)  images 
of  this  target,  when  front  illuminated  in  a  tank  of  water  with  a  small  amount  of  milk  added,  are 
presented  in  Figs.  1A  and  IB,  where  the  PS  and  PD  image  intensities  were  scaled  using  an  affine 
transformation  to  utilize  the  full  dynamic  range  of  the  8-bit  display.  In  Figs.  1C  and  ID  we 
present  the  PD  images  of  the  same  target,  but  in  these  images,  we  have  used  two  different 
threshold  cuts  in  the  histogram  of  the  PD  signals.  Specifically,  after  computing  the  histogram 
of  the  image,  the  brightest  and  darkest  intensity  bins  with  pixel  population  less  than  a 
certain  percentage  (e.g.,  T  =  1%  for  Fig.  1C  and  T  =  10%  for  Fig.  ID)  of  the  population 
of  the  largest  bin  are  deemed  as  outliers  and  assigned  the  brightest  and  darkest  display 
intensity  values  (0  or  255  for  a  common  8-bit  display).  The  rest  of  the  image  has  been 
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linearly  scaled  to  cover  the  full  dynamic  range  of  the  display.  As  we  can  see  from  these 
figures,  the  patches  and  particularly  the  difference  between  their  various  directions  of  abraded 
lines  can  be  easily  detected  in  the  PD  images  —  much  more  efficiently  than  in  the  PS  image. 
These  PD  techniques  can  remarkably  increase  the  target  feature  visibility  and  enhance  the 
contrast.  In  our  efforts  here,  we  have  clearly  demonstrated  that  bio-inspired  polarization 
techniques  can  be  instrumental  in  target  feature  detection  and  visibility  enhancements.1 


Fig.  1.  Affine-transformed  Polarization-Sum  (PS)  and  Polarization-Difference  (PD)  images  of  a  7-patch  aluminum  disk,  front- 
illuminated  in  a  tank  of  water  to  which  a  small  amount  of  milk  is  added.  Panel  A  shows  the  affine-transformed  of  PS  image,  which  is 
effectively  equivalent  to  a  conventional  image,  whereas  Panels  (B),  (C),  and  (D)  present  the  various  PD  images,  using  T  =  0  %,  T  = 
1%,  and  T  =  10%  thresholding  in  the  PD  histograms. 


In  our  earlier  work,  we  had  also  shown  that  the  PD  technique  can  increase  the  distance 
over  which  a  target  can  be  detected  in  scattering  media  [24],  Using  the  statistical-analytical 


1  This  target  was  specifically  constructed  in  order  to  make  the  light  scattered  from  the  patch  areas  partially  polarized  (with  the  major 
axis  of  the  partial  polarization  parallel  to  the  direction  of  abrasion),  while  the  light  scattered  from  the  sandblasted  areas  (including  the 
7lh  patch)  is  largely  unpolarized.  (As  a  result,  in  the  PD  image  of  this  7-patch  target,  Figs  IB,  1C,  and  ID,  the  centered  7lh  patch  is 
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methods  of  signal  detection  theory,  we  analyzed  the  intrinsic  detectibility  of  targets  in  scattering 
media  when  polarization-difference  imaging  is  utilized  [24],  Figure  2  shows  the  spatial 
sensitivity  index  computed  at  each 
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Fig.  2.  Spatial  sensitivity  index  of  the  variable  da  ,  computed  at  each  effective  distance  for  both  types  of  images  [24],  This  quantity 

reveals  how  well  one  can  distinguish  the  patch  from  its  surrounding.  Panel  A  presents  the  data  from  the  left  target  patch;  panel  B 
presents  the  data  from  the  right  target  patch.  At  all  times,  PD1  performs  as  well  or  better  than  the  conventional  (PS)  system.  In 
addition,  the  PS  images  degrade  faster  as  a  function  of  the  attenuation  length  (proportional  to  the  scatterer  concentration).  The 
exponential  curves  fitted  to  the  data  indicate  that  if  any  particular  value  of  spatial  sensitivity  index  is  chosen  as  the  threshold  of 
detectibility,  then  the  PD  system  will  reach  this  level  at  distances  at  least  double  that  of  conventional  (PS)  systems. 


effective  distance  for  bdth  types’ of  images  (P’S  ahd'FD  images)  of'the  right  add  left  patches.  This 
index  is  a  measure  of  how  well  the  patch  can  be  reliably  distinguished/detected  from  its 
surrounding  in  these  images  by  an  “ideal  observer”.  Figure  2  clearly  shows  that  at  low  effective 
target  distances  the  statistical  distributions  of  sensitivity  index  of  the  patches  and  the  surrounding 
frames  are  at  least  as  separated  in  the  PD  images  as  they  are  in  the  PS  images.  However,  the 
separation  of  these  distributions  decays  more  slowly  in  the  PD  case  than  in  the  PS  case  as 
effective  distance  is  increased.  If  any  of  the  value  of  the  spatial  sensitivity  index  is  chosen  as  a 
threshold  of  detectibility,  this  value  is  reached  in  the  PD  images  at  effective  lengths  2-3  times 
greater  than  in  the  PS  images.  This  reveals  the  enhanced  intrinsic  detectibility  of  PD  images,  and 
offers  great  potential  for  enhancement  of  underwater  observation,  seeing  through  fog,  and  sensing 


indistinguishable  from  the  sandblasted  surface  of  the  disk,  and  thus  only  six  abraded  patches  can  be  seen.)  For  detailed  description  of 
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in  other  scattering  environments.  More  results  can  be  found  in  our  manuscripts  shown  in  the 
appendix. 

2. Bio-Inspired  Adaptive  Algorithms  in  Polarization  Sensing  and  Imaging 

One  of  the  important  issues  in  polarization  imaging  is  how  to  process  the  polarization 
information  after  (or  while)  it  is  collected  by  the  imaging  system.  As  one  possible  processing 
technique,  as  shown  in  the  previous  section  we  have  worked  on  the  methods  of  “polarization- 
difference  imaging”  (PDI),  inspired  by  polarization  vision  in  certain  species,  and  we 
demonstrated  that  optical  imaging  systems  utilizing  PDI  techniques  may  facilitate  the  detection  of 
targets  in  scattering  media,  even  when  the  targets  produce  only  very  weak  polarization.  In  that 
work,  we  showed  that  in  the  PDI  technique,  the  intensities  of  the  two  orthogonal  polarization 
components  of  imaging-forming  light  for  each  pixel  at  (x,_y),  i.e.,  I^x,y)  and  IL(x,y),  were 

captured  and  then  the  “polarization-sum”  (PS)  Ips  (x,y)  =  In  (x,y)  +  IL  and 

“polarization-difference”  (PD)  I PD  (x,jy)  =  In  (x, y)~Ix  {x»y)  intensities  were  formed.  It  had 

also  been  shown  by  one  of  the  alumni  of  our  research  team,  Scott  Tyo  [28],  that  the  optimum 
linear  combination  of  In(x,y)  and  Ix(x,y)  channels  for  a  scene,  in  which  the  polarization  angle  is 

assumed  to  be  a  random  variable  with  a  uniform  probability  density  function  over  the  image 
pixels,  are  indeed  PS  and  PD  signals,  utilizing  the  analogy  and  parallelism  with  the  biology  of 
color  vision  in  the  human  visual  system  and  related  principal  component  analysis  done  by 
Buchsbaum  and  Gottschalk  [37].  Therefore,  in  that  scenario  the  PS  and  PD  channels  were 
optimum  in  the  information  theoretic  sense,  i.e.,  their  contents  were  statistically  uncorrelated, 
requiring  minimum  bandwidtfo-for  ■-encoding'  [2-8].  However,  the'  situation,  is  different  when  the 
scene  has  a  polarization  distribution  with  a  non-uniform  probability  density  for  the  state  of 
polarization,  such  as  occurs  in  shallow  water  where  mostly  the  sunlight  penetrating  the  water  is 
partially  horizontally  polarized  as  viewed  by  the  polarization-sensitive  marine  species.  What 
would  then  be  the  optimum  linear  combination  for  polarization  channels  in  this  general  case?  We 
have  been  investigating  techniques  to  adaptively  form  the  optimum  linear  combination  for 
polarization  channels.  Specifically,  the  two  linear  channels  in  general  can  be  expressed  as 

(X,y)  =  alu  {x,y)+P  IL  (x,y)  (1) 

pcf*  (x,y)  =  r  h  (x>y) +C 1  A*,y)  (2) 


the  experimental  setup,  see  Ref  [24]. 
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with  unequal  weighting  coefficients  a ,  /? ,  y,  and  £ ,  which  should  be  determined  based  on  the 
statistics  of  the  polarization  distribution  in  any  given  scene.  Utilizing  the  technique  of  principal 
components  analysis  for  non-uniform  distributions  of  polarization  state  in  an  image,  we  can 
determine  the  appropriate  values  for  these  coefficients,  which  will  depend  on  the  polarization 
statistics  of  the  scene.  Moreover,  once  an  algorithm  for  determining  and  applying  them  is  in 
place,  the  coefficients  can  be  adaptively  adjusted  as  the  environment  observed  by  the  imaging 
system  changes  (for  example,  due  to  the  transit  of  the  sun  in  the  sky)  or  as  the  imaging  system  is 
moved  to  different  environments.  Such  optimum  combinations  for  polarization  channels  with 
unequal  weighting  coefficients  suitable  for  environment  with  non-uniform  polarization 
distribution  (such  as  under  water)  may  point  to  an  interesting  processing  in  the  polarization  vision 
in  certain  aquatic  species,  and  can  lead  to  images  with  higher  contrast  and  better  target  detection 
for  man-made  imaging  systems.  We  have  been  working  on  the  theoretical  and  experimental 
aspects  of  such  adaptive  algorithms  [34].  Figure  4  shows  a  preliminary  sample  of  the  results  of 
our  work  in  this  bio-inspired  adaptive  algorithm.  The  experimental  setup  is  shown  in  Fig.  3.  In 
this  case,  the  7-patch  disk  is  immersed  in  a  tank  of  water  with  a  small  amount  of  milk  added,  and 
the  tank  is  illuminated  from  the  left  side  and  the  images  are  taken  by  a  digital  camera  located  on 
top  of  the  tank  looking  into  the  tank  from  the  above.  Since  the  light  is  coming  from  the  side  of 
the  tank,  the  background  light  scattered  from  the  milky  water  and  propagating  towards  the  camera 
is  partially  polarized,  with  orientation  of  polarization  along  the  up-down  direction  in  the  scene 
viewed  by  the  camera.  Images  are  then  collected  at  three  orientations,  0  =  0 ,45  ,90'  of  the 
linear  polarization  analyzer  in  front  of  the  camera,  and  then  the  polarization  information  at  each 
pixel  is  evaluated.  We  then  analyzed  all  possible  combinations  for  the  two-channel  polarization 
imaging  systems  as  a : function- of  channel  orientations  0,  and  02  (not  shown  here),  and  then  for 
each  pair  of  two  channels,  using  the  principal  component  analysis  we  calculated  the  weighting 
coefficients  a(0],02 ) ,  /?  ( 0, ,  02 ) ,  y(0, ,02 ) ,  ^(0,,02)  to  form  the  principal  component  (PC) 

outputs  as  PCI  =  cr(0, ,02 )/, (0, )  +  /?(0, ,02 )/2 (02 )  and  PC2  =  y(0l,92)ll(0l)  +  £(0l,O2)l2(02) . 
Since  the  background  light  coming  from  the  water  around  the  target  is  not  unpolarized,  we  have 
found  from  our  analysis  that  the  coefficients  a,(0,,02),  /?(0,,02),  /(0i,02),  ^(0,,02),  which  are 

determined  from  eigenvectors  of  the  covariance  matrix  of  two-channel  polarization  signals,  are 
not  identical,  (i.e.,  a  *  and  y  *  -£ ,  but  they  follow  the  relations  y=  J3  and  £  =  -a )  and  thus 
the  PC  1  and  PC2  are  different  from  “regular”  polarization-sum  and  polarization-difference 
signals.  Figure  4A  and  4B  show  the  “regular”  PS  and  PD  images  of  the  scene,  obtained  using  the 


Page  9 


expressions  PS  =  /,  (90  )  + 12  (o  )  and  PS  =  /,  (90  )-I2  (0‘ ) ,  after  they  are  affine  transformed 

with  1%  thresholding,  whereas  Figs.  4C  and  4D  present  the  same  scene  but  with  the  adaptive  PC 
signals  (i.e.,  PCI  and  PC2,  respectively)  obtained  using  the  adaptive  analysis  we  described 
above.  If  we  compare  Fig.  4B  and  4D,  we  can  see  that  in  the  PC2  image  (i.e.,  4D)  the  effect  of 
partially  polarized  background  is  reduced,  resulting  in  a  better  contrast  and  visibility  for  the  target 
features  [34].  This  suggests  various  interesting  possibilities  for  polarization  imaging  in  changing 
environments  where  the  optimal  linear  combinations  of  the  two-channel  (or  three-channel) 
polarization  should  be  adaptively  adjusted  for  better  target  detection.  This  is  a  promising 
technique,  and  we  are  currently  developing  and  expanding  this  technique.  More  results  can  be 
found  in  our  manuscripts  shown  in  the  appendix. 


Fig.  3.  Schematic  of  the  experimental  setup  for  the  sample  results  shown  in  Fig.  4  for  the  study  of  adaptive  polarization  imaging. 
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Fig.  4.  Panels  A  and  B  show  the  “regular”  PS  and  PD  images  of  the  7-patch  disk  as  a  target  in  Fig.  3,  after  they  are  affine  transformed 
with  1%  thresholding.  Panel  C  and  D  present  the  principal  components  1  and  2,  PCI  and  PC2,  for  the  “optimal”  choice  of  the  channel 
orientations  and  weighting  coefficients  in  this  example.  Panel  4D  clearly  shows  a  better  visibility,  feature  extraction,  and  lesser  effect 
from  the  background  light  as  compared  with  Panel  4B.  More  detailed  results  can  be  found  in  our  manuscript  that  is  being  prepared 
for  submission  for  publication.  See  the  appendix  for  a  draft  of  this  manuscript. 


3. Bio-Inspired  Ideas  for  Non-Invasive  Detection  and  Imaging 'of  Latent  Finger  Prints  and  Skin 
Surface 

One  of  the  interesting  species  with  polarization  vision  is  backswimmer  flying  water  bug, 
Notonecta  glauca.  It  has  been  found  by  Schwind  [11-13]  that  in  the  ventral  region  of  the 
compound  eyes  of  this  species  the  microvillar  directions  of  the  photoreceptors  are  arranged  such 
that  they  can  pick  up  horizontally  polarized  light  very  well.  This  ability  helps  the  species  to  find 
bodies  of  water  in  their  habitats,  since  the  light  reflected  from  the  surface  of  the  water  is  mostly 
horizontally  polarized.  Furthermore,  as  discovered  by  Schwind  [11-13],  this  species  lands  on  the 
surface  of  the  water  at  angle  of  53°  from  the  normal  [5],  which  is  approximately  the  Brewster’s 
angle  of  water,  therefore  optimizing  the  horizontal  polarization  coming  to  its  visual  system  [11, 
5],  (See  Fig.  5).  So  this  species  can  “see”  the  surface  of  the  water  efficiently  and  thus  can  land 
on  it.  Fig.  6  shows  the  polarimetric  images  of  a  body  of  water  and  a  tree,  obtained  using  a  digital 
camera  equipped  with  a  polarization  analyzer,  after  having  been  processed  in  order  to  find  the 
pixel  intensity  distribution  (Panel  A),  degree  of  linear  polarization  (Panel  B)  and  the  angle  of 
polarization  orientation  (Panel  C)  for  each  pixel.  As  we  can  see  from  Panels  B  and  C,  the  surface 
of  the  water  is  highly  polarized  in  the  horizontal  direction. 


Inspired  by  this  ability  in  the  backswimmer  water  bug,  we  have  been  motivated  to 
explore  how  we  can  take  advantage  of  polarization  in  detecting  other  “smooth  surfaces”  and/or 
“minute  deformation”  on  smooth  surfaces.  We  decided  to  explore  the  possibility  of  using  such 
bio-inspired  techniques  in  detecting  latent  finger  prints  on  smooth  surfaces.  After  all,  finger 
prints  can  be  regarded  as  “deformation”  of  an  otherwise  smooth  surface. 
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Fig.  5.  Backswimmer  flying  water  bug  whose  visual  system  is  sensitive  to  the  horizontal  polarization.  Schwind  [11-13]  discovered 
that  this  species  lands  on  the  surface  of  the  water  at  angle  53*  from  the  normal,  which  is  approximately  the  Brewster  angle  of  water, 
therefore  optimizing  the  horizontal  polarization  coming  to  its  visual  system. 
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cotton  cloth  lining,  taken  with  the  digital  camera  without  any  polarization  information  (the 
image  is  auto-level  enhanced.)  There  is  a  finger  print  on  this  CD  cover,  which  is  very  hard  to 
detect  in  the  regular  image  shown  in  Panel  7A.  Then  a  polarization  analyzer  was  added  to  the 
digital  camera,  and  then  the  complete  polarization  information  (i.e.,  the  intensity,  the  degree  of 
linear  polarization,  and  the  angle  of  polarization  orientation  for  each  pixel)  was  derived.  Panel 
7B  shows  the  image  of  the  “degree  of  linear  polarization”  after  histogram-equalization.  This 
image  is  not  the  pixel  intensity  image,  but  rather  is  the  distribution  of  the  degree  of  linear 
polarization  of  pixels.  So  we  can  clearly  see  the  presence  of  the  finger  print  on  this  cover.  Panels 
C  and  D  present  the  “cropped”  version  of  Panels  A  and  B,  after  they  are  auto-level  enhanced  and 
histogram-equalized,  respectively.  We  can  see  that  even  after  we  realize  the  location  of  the  finger 
print,  the  “regular”  image  in  Panel  C  is  less  detectable  than  the  one  in  Panel  D,  which  is  from  the 
degree  of  linear  polarization.  Panel  E  represents  the  segmentation  of  the  degree  of  linear 
polarization,  and  then  Panel  F  shows  the  extracted  segment  with  the  finger  print.  Finally,  Panel  G 
illustrates  the  application  of  Canny  edge  detection  [61]  to  the  selected  segment  in  Panel  F.  This 
shows  reasonably  well  the  lines  of  the  finger  prints  in  our  image.  We  are  currently  working  on 
various  techniques  to  improve  this  bio-inspired  technique  for  finger  print  detection.  More  results 
can  be  found  in  our  manuscript  that  has  been  submitted  for  publication,  and  is  attached  in  the 
appendix. 


We  will  also  consider  applying  this  technique  to  detection  of  other  surface  deformation 
and  variation. 


(A)  (B) 
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(G) 

Fig.  7.  Preliminary  results  of  a  bio-inspired  idea  for  non-invasive  imaging  of  a  latent  finger  print  on  a  smooth  surface.  Panel  A  presents  a  normal 
(auto-level  enhanced)  image  of  a  soft  plastic  CD  protector  with  white  cotton  cloth  lining,  taken  with  the  digital  camera  without  any  polarization 
information.  Panel  B  shows  the  image  of  the  “degree  of  linear  polarization”  after  it  is  histogram-equalized.  This  image  is  not  the  pixel  intensity 
image,  instead  it  is  the  distribution  of  the  degree  of  linear  polarization  of  pixels.  The  finger  print  On  this  cover  is  clearly  visible-.  Panels  C  and  D 
present  the  “cropped”  version  of  Panels  A  and  B,  after  they  are  auto-level  enhanced  and  histogram-equalized,  respectively.  Panel  E  represents  the 
segmentation  of  the  degree  of  linear  polarization,  and  Panel  F  shows  the  segment  extraction  with  the  finger  print  selected.  Panel  G  illustrates  the 
application  of  Canny  edge  detection  to  the  selected  segment  in  Panel  F.  More  detailed  results  can  be  found  in  our  manuscript  that  has  been  submitted 
for  publication.  This  is  attached  in  the  appendix. 


Fig.  8  shows  a  preliminary  result  we  obtained  from  polarization  imaging  from  the  skin  of  the 
back  of  a  hand.  Panel  B  shows  the  “degree  of  linear  polarization”  of  the  skin  (after  histogram 
equalization),  which  may  reveal  more  information  about  the  surface  of  the  skin.  We  will 
investigate  a  number  of  specific  parameters  in  such  imaging  of  skin,  and  explore  how  the 
angles  of  illumination  and  viewing  (and  the  combination)  affect  the  images. 
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(A)  (B) 

Fig.  8.  Preliminary  results  of  polarization  imaging  of  skin.  Panel  A  shows  a  “regular”  image  of  the  skin  of  back  of  a  hand  without 
any  polarization  information.  Panel  B  shows  the  “degree  of  linear  polarization”  of  the  skin  after  histogram  equalization. 


4. Polarization  Shadow  Removal:  Polarization  Imagine  Techniques  for  Detection  of  Objects 
Hidden  in  Shadows 

In  the  course  of  our  study  of  bio-inspired  polarization  imaging  of  natural  and  man-made 
scenes,  we  have  noticed  that  when  the  distribution  of  the  “degree  of  linear  polarization”  (instead 
of  regular  pixel  intensity  distributions)  at  each  pixel  are  shown,  certain  aspects  of  the  scenes  such 
as  “shadows”  are  presented  differently.  In  fact,  we  have  found  that  in  certain  situations,  when  the 
image  of  the  degree  of  polarization  is  shown,  the  regular  shadows  are  either  disappeared  or  have 
been  less  noticeable.  We  show  our  preliminary  results  of  such  “shadow  removal”  by  the 
polarization  imaging  in  Fig.  9.  Panel  A  presents  the  regular  intensity  image  of  a  scene  on  the  roof 
of  our  building  (after  histogram  equalization),  while  Panel  B  shows  the  distribution  of  the  degree 
of  linear  polarization  of  the  same  scene  (after  it  is  histogram  equalized).  We  notice  several 
interesting  features  in  this  image:  The  shadow  in  front  of  the  chimney,  which  is  clearly  observed 
in  Panel  A,  is  no  longer  there  in  Panel  B.  Likewise,  the  shadow  beneath  the  edge  of  the  roof  is 
effectively  “removed”  in  Panel  B.  Therefore,  if  there  is  an  object  hidden  in  these  regions  of  the 
image  in  Panel  A,  it  would  become  “visible”  in  Panel  B.  This  can  be  seen  when  we  look  at  the 
details  of  the  “object”  to  the  right  of  the  chimney  -  such  details  cannot  be  seen  in  Panel  A  since 
the  shadow  makes  it  dark  to  see  these  details,  while  in  Panel  B  these  are  clearly  more  “visible”. 
Moreover,  the  strings  holding  the  chimney  are  more  visible  in  Panel  B  because  the  light  reflected 
from  them  is  partially  polarized.  Furthermore,  the  sky  background  has  one  polarization,  distinct 
from  the  clouds,  and  the  clouds  now  “pop-out”  with  much  better  contrast.  If  an  airplane  is  flying 
through  the  clouds  (but  not  too  deep  into  them),  it  may  also  pop  out.  All  these  features  can  offer 
an  exciting  possibility  for  detection  of  objects  hidden  in  shadow  regions  using  the  bio-inspired 
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polarization  imaging  techniques.  Furthermore,  this  can  alter  the  perception  of  object  boundaries, 
if  the  image  is  taken  and  shown  using  this  technique.  More  results  can  be  found  in  our 
manuscript  that  is  being  prepared  for  submission  for  publication  and  is  attached  in  the  appendix. 


(A)  (B) 

Fig.  9.  Images  of  a  scene  on  the  roof.  Panel  A  shows  the  “conventional”  pixel  intensity  image  (after  histogram  equalization),  while 
Panel  B  presents  the  distribution  of  the  “degree  of  linear  polarization”  for  pixels  in  the  scene  (after  it  is  histogram  equalized).  Note 
that  some  of  shadows  in  Panel  A  have  been  effectively  “disappeared”  in  Panel  B,  and  as  a  result  certain  details  can  be  “seen”  in  Panel 
B.  (see  the  text  and  the  appendix  for  more  detail  and  other  images.) 


5. Bio-Inspired  Display  of  Polarization  Information  Using  Selected  Visual  Cues 

As  has  been  shown  in  the  previous  parts,  it  is  clear  that  the  polarization  of  the  image¬ 
forming  light  carries  important  information  about  the  scene  that  is  being  imaged.  However,  an 
interesting  issue  arises  in  dealing  with  representing  the  polarization  information  obtained  from  a 
polarimetric  imaging  system.  Since  the  unaided  human  eye  cannot  directly  “see”  polarization 
information,  how  can  then  such  polarization  information  that  is  obtained  by  a  polarimetric 
imaging  system  be  presented  to  a  human  observer?  After  all,  the  outputs  of  any  imaging  system 
must  be  presented  in  such  a  manner  that  can  be  interpretable  and  recognizable  by  a  human 
observer.  What  this  implies  is  that  such  information  should  be  mapped  into  visual  cues  that  can 
be  easily  detectable  by  a  human  observer,  hopefully  without  altering  the  other  information  carried 
by  cues  such  as  spectral  and  luminance  information.  Effectively,  some  form  of  “sensory 
substitution”  is  required  for  presenting  polarization  “signals”  to  a  “polarization-blind”  observer, 
without  affecting  other  visual  information  such  as  color  and  brightness.  In  our  group’s  earlier 
work,  we  had  shown  a  bio-inspired  mapping  for  the  cases  where  the  original  spectral  information 
(i.e.,  the  perceived  colors)  of  the  scenes  was  not  of  interest  [26],  In  that  mapping,  polarization- 
difference  information  was  pseudo-color-coded,  based  on  the  opponent-colors  model  of  human 
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vision,  resulting  in  a  contrast  enhancement  of  polarization  imagery  of  objects  in  scattering  media 
such  that  the  objects  became  easily  detectable  by  a  human  observer  [26],  However,  that  mapping 
ignored  the  spectral  information  in  the  original  scene  —  information  normally  mapped  into 
perceived  color.  In  order  to  preserve  the  spectral  and  luminance  information  in  an  image,  we 
need  to  map  the  polarization  into  visual  cues  other  than  the  color  and  brightness.  The  human 
visual  system  detects  and  encodes  a  great  variety  of  visual  cues.  These  include  color,  intensity, 
coherent  motion,  texture,  flicker,  to  name  a  few.  In  the  first  Phase  of  our  efforts,  we  have  been 
conducting  research  on  a  variety  of  bio-inspired  representational  schemes  as  well  as  certain 
fusion  techniques  in  order  to  present  polarization  information  to  the  human  observer  in 
meaningful  cues,  that  is,  in  a  form  perceptible  to  human  vision  [30-34].  Here,  we  briefly  mention 
some  of  our  results.  The  details  can  be  found  in  our  published  works  [30-32],  These  techniques 
include: 

(a)  Coherently  moving  dots:  Since  human  vision  is  highly  sensitive  to  motion  —  including 
coherent  motion  detection,  form  from  motion,  and  biological  motion  [39-42]  ---  the  sensitivity  of 
coherence  detection  in  human  eye  can  be  exploited  to  display  polarization  information  in  an 
image.  Wc  have  used  coherently  moving  dots  superimposed  on  images  to  represent  various 
polarization  parameters  [30-31],  Fig.  10  illustrates  a  sample  of  our  results. 


Fig.  10.  Panels  (A)  and  (C):  Superimposing  moving  dots  on  the  PS  images  of  the  2-patch  and  7-patch  targets  using  the  contrast  scheme 
[30].  Panels  (B)  and  (D):  corresponding  images  with  the  percentage  scheme  [30],  Dot  size  is  lxl  pixel  in  Panels  (A)  and  (B),  and  2x2 
pixels  in  Panels  (C)  and  (D)  Since  here  we  need  to  show  “static  image”,  some  “arrows”  have  been  added  to  give  indication  of  the 
direction  of  motion  of  the  coherently  moving  dots.  We  have  made  movie  files  showing  the  coherent  motion  of  these  dots.  Movies  are 
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available  in  our  paper  in  the  online  journal  Optics  Express  [30]. 

(b)  “Temporal  fusion”:  We  utilize  the  dimension  of  time,  as  an  additional  degree  of  freedom,  to 
dynamically  superimpose  polarization-based  images  with  non-polarized  images  in  accordance 
with  the  established  sensitivity  of  human  eye  to  temporal  variation  of  brightness  and  color  [31]. 
A  variety  of  image  fusion  techniques  have  been  used  by  many  researchers  to  combine  two  or 
more  images  from  different  modalities  (such  as  images  from  the  visible  band  and  the  IR  band) 
into  a  single  image  that  ideally  contains  all  the  important  information  of  original  images  (see  e.g., 
[43-47]).  In  principle,  those  fusion  techniques  can  also  be  used  to  fuse  polarization  images  with 
non-polarization  images  (e.g.,  intensity-only  images).  We  have  introduced  the  concept  of 
“temporal  fusion”,  where  the  fusing  of  two  different  images  is  achieved  in  time  domain  by  taking 
advantage  of  the  biological  sensitivity  of  human  visual  system  to  the  "fading  in  and  fading  out" 
between  two  image  modalities  [31].  As  applied  to  polarization,  the  image  modalities  of  interest 
here  are  an  intensity-only  image  (i.e.,  PS  image)  and  the  polarization-difference  (PD)  information 
of  the  same  scene  (e.g.,  this  latter  image  could  be  a  grayscale  representation  of  the  PD  data,  or  a 
pseudo-color  representation).  A  dynamic  image  (i.e.,  a  movie)  consisting  of  a  gradual  periodic 
transition  from  a  PS  image  to  PD  image  is  then  created,  with  the  frequency  of  transition  chosen 
such  that  the  human  eye  can  easily  and  most  sensitively  detect  this  fading  in  and  fading  out.  In 
this  way,  the  observer  can  clearly  see  and  distinguish  the  PS  information  and  PD  information  in 
the  same  dynamic  image.  In  other  words,  we  take  advantage  of  the  dimension  of  “time”  in  order 
to  fuse  information  from  two  different  modalities,  i.e.,  from  polarization  and  non-polarization 
information.  If  the  PD  image  is  formed  as  a  pseudo-colored  image,  then  we  choose  the  frequency 
of  fading  between  the  PS  and  PD  image  in  accordance  with  sensitivity  of  color  channels  in  the 
human  eye.  Figure  1 1  shows  a  few  snapshots  of  one  of  our  movies  for  such  temporal  fusion. 
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frame  15 

frame  18 

frame  25 

frame  30 

Fig.  11.  Snapshots  of  temporal  fusion  of  the  grayscale  PS  and  PD  images  of  the  back-illuminated  7-patch  target  using  the  “sinusoidal” 
modulation  [31],  Every  complete  cycle  of  the  movie  file  includes  30  frames,  which  can  be  viewed  with  the  rate  of  30  frames/sec.  In 
this  figure,  we  only  present  frames  No  1,  5,  10,  12,  15,  18,  25,  and  30,  from  the  left  to  right  and  the  top  to  bottom.  The  movie  files  are 
available,  and  we  can  send  them  to  the  interested  reader. 


We  have  also  studied  the  temporal  fusion  of  a  color  image  of  a  natural  scene  with  its  pseudo¬ 
color  polarimetric  image.  It  is  well  known  that  for  the  light  coming  from  every  point  in  a  scene, 
three  parameters  related  to  the  polarization  information  detectable  by  a  2-D  imaging  sensor,  in 
which  the  phase  information  between  the  two  orthogonal  polarization  components  is  not 
available,  can  be  described  as  the  energy  intensity  (or  brightness)  1 ,  degree  of  linear  polarization 
p ,  and  orientation  angle  cpo  of  the  polarization  ellipse  of  the  polarized  portion  of  the  light  from 
that  point.  The  value  of  p  varies  between  0  and  1  where  p  =  0  presents  unpolarized  light  and 
p  =  1  corresponds  to  a  completely  linearly  polarized  signal.  To  find  these  parameters  at  every 
pixel,  one  needs  to  collect  a  minimum  of  three  images  of  the  same  scene;  each  image  taken  with  a 
different  orientation  of  the  polarization  analyzer  in  front  of  the  camera,  e.g.,  with  0” ,  45  ,  and  90" 
orientation  of  the  polarization  analyzer.  These  three  independent  “measurements”  can  provide 
one  with  three  equations  from  which  the  parameters  I ,  p,  and  <po  can  be  evaluated  for  each 
pixel.  To  display  these  three  parameters  of  a  scene  in  a  single  image,  one  can  use  a  variety  of 
mapping  strategies.  We  have  investigated  several  different  mapping  possibilities.  One  such 
mapping  is  the  Bernard  and  Wehner  mapping  [48],  who  suggested  the  mathematical  similarities 
between  a  theoretical  3-parameter  polarization  vision  system  and  the  widely  used  3-parameter 
color  vision  system.  They  also  pointed  out  that  some  species  may  indeed  have  3-parameter 
polarization  vision  system  [48],  The  main  analogy  they  draw  is  the  parallelism  of  the  intensity  I 
with  the  value  V,  the  degree  of  polarization  p  with  the  saturation  S,  and  the  angle  <pa  with  the 
hue  H  (in  the  HSV  color  space).  This  parallelism  has  been  used  in  the  past  in  certain 
polarization-sensitive  imaging  systems  [49-51],  Here,  as  an  input  to  our  temporal  fusion 
algorithm,  we  utilize  this  pseudo-color  mapping  based  on  this  parallelism  to  produce  a  pseudo¬ 
color  polarimetric  image.  For  display  purpose,  we  incorporate  the  image  enhancement  technique 
“histogram  equalization”  on  the  original  value  V  (in  the  HSV  space)  of  the  “true-color”  image, 
and  use  this  equalized  V  as  the  V  channel,  and  the  contrast  enhanced  p  data  for  the  saturation 
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channel  S.  For  the  angle  of  polarization  <po ,  we  map  2 <po  to  the  hue  angle.  As  an  example,  we 
use  a  picture  of  flowers  and  leaves  which  we  have  taken  outdoors  by  taking  three  images  with 
three  orientations  of  the  polarization  analyzers  as  mentioned  above.  A  processed  “true-color” 
image  of  this  scene  is  shown  in  Fig.  12A  after  the  histogram  equalization  has  been  applied;  the 
distribution  of  degree  of  linear  polarization  p(x,y) ,  after  contrast  enhancement,  is  shown  in  Fig. 
12B;  and  the  distribution  of  the  angle  of  polarization  ellipse,  <po(x,y) ,  is  depicted  in  Fig.  12C  with 
the  angle-to-hue  color  wheel  shown  at  its  right.  Figure  12D  illustrates  the  mapping  of  {I,p,<po) 
into  ( V,S,H ).  We  notice  that  the  images  with  polarization  information  (Figs.  12B,  12C,  and 
12D)  reveal  certain  additional  information  not  available  in  the  regular  color  image  in  Fig.  12 A. 
For  instance,  we  note  that  in  the  distribution  of  p{x,y)  shown  in  Fig.  12B  the  leaves  in  general 
exhibit  higher  degree  of  linear  polarization  than  the  flowers  (the  flowers  look  “darker”  than  the 
leaves  in  p{x,y) ).  Moreover,  we  observe  that  certain  parts  of  the  scene  with  low  intensity  shown 
in  the  conventional  image  in  Fig.  12  A,  such  as  the  little  leaf  in  the  foreground  slightly  below  the 
three  large  leaves,  may  appear  more  noticeable  in  the  images  of  p(x,y)  and  <po(x,y)  shown  in 
Figs.  12B  and  12C,  since  the  degree  of  polarization  at  a  pixel  may  in  general  vary  independently 
of  intensity  of  that  pixel.  We  have  applied  our  temporal  fusion  technique  to  fuse  the  image 
shown  in  Fig.  12A  with  that  in  Fig.  12D  and  to  produce  movie  files  for  such  fusion.  Figure  13 
illustrates  the  snapshots  of  this  temporal  fusion  with  sinusoidal  modulation  [31]. 
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(C)  (D) 

Fig.  12:  (A)  A  processed  “true  color”  image  of  a  scene  after  the  histogram  equalization  of  its  V  channel;  (B)  Degree  of  linear 
polarization  p(x,y)  image  after  it  is  contrast  enhanced;  (C)  Angle  of  polarization  ellipse,  <P0(x,y),  shown  in  pseudo-color 

scheme.  The  angle-to-hue  color  wheel  is  shown  to  the  right;  and  (D)  Pseudo-color  mapping  of  three  parameter  (/,  p,  (fl0  )  into 

(V,S,H). 


Fig.  13.  Snapshots  of  temporal  fusion  of  Fig.  12A  (color  image  of  the  scene)  with  Fig.  12D  (pseudo-color  polarimetric  image  of  the 
same  scene)  using  the  sinusoidal  function  CC(n)  [31].  Every  complete  cycle  of  the  movie  file  includes  30  frames,  played  back  at  the 
rate  of  30  frames/sec.  The  movie  files  are  available  upon  request. 


(c)  Flicker:  In  this  method  of  presenting  polarization  information,  we  have  exploited  the 
sensitivity  of  human  eye  to  intensity  flicker  in  an  image  [52].  After  forming  the  PS  and  PD 
images  of  a  scene,  we  intend  to  bring  the  observer’s  attention  to  the  regions  of  the  image  having 
PD  signals  with  opposite  signs.  We  find  these  regions  by  searching  for  pixels  whose  raw  PD 
values  are  greater  than  +  8  and  those  that  are  less  than  -8 ,  where  8  is  a  given  threshold.  These 
regions  we  name  (+y**ancP (-)  regions,  respectively.  The  rest  of  the  image,  in  which 
-8  <  PD  <  8 ,  is  the  “PD  neutral”  region.  Then  we  temporally  modulate  the  intensity  of  the  pixels 
in  PS  image  belonging  to  the  (+)  and  (-)  regions  from  frame  to  frame,  forming  a  movie  file. 

Such  temporal  modulation  can  be  expressed  as:  7W  (x,y,n)  =  TPS  [„/(*, y)](l  +  M+  sin (&/„)) , 
/H(x,y,n)  =  rra[„/(.x,y)](l  +  M_sin(D/„  +zr)).  Here,  x  and  y  are  coordinates  of  the  pixels 

within  the  image,  n  is  the  frame  number  ranging  from  1  to  N  where  N  is  the  number  of  frames 
in  a  complete  cycle,  tn  corresponds  to  the  time  of  frame  n ,  M+  and  M  are  the  modulation 
coefficients  for  the  (+)  and  (-)  regions,  respectively,  Q.  =  2jiF  where  F  is  the  frequency  of 
flicker  in  Hz.  There  is  180“  phase  difference  between  the  flickers  in  the  (+)  and  (-)  regions.  We 
have  applied  such  a  technique  in  several  images  obtained  using  the  polarimetric  imaging.  The 
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images  can  be  found  in  our  paper  [3 1  ]  and  the  movies  are  available  upon  request. 

(d)  Textures:  In  human  visual  perception,  it  is  known  that  certain  forms  of  textures  can  be 
segregated  more  easily  than  others  [53-56].  Therefore,  textures  can  be  another  set  of  visual  cues 
that  can  be  used  to  display  polarization  information.  We  have  been  exploring  these  ideas  for 
polarization  visualization.  In  order  to  utilize  textures  for  this  purpose,  we  first  subdivide  the 
entire  image  into  small  cells  of  mxm  pixels  (e.g.,  m  =  10),  and  we  evaluate  the  average  PS  and 
PD  values  in  each  cell.  We  then  identify  the  (+)  and  (-)  regions  of  the  image  which  correspond 
to  pixels  with  (PD)cell  >  +5  and  < PD)CC„  <  - 8 ,  as  was  described  above.  Here  (PD)cdl  denotes  the 
average  value  of  PD  in  each  cell.  We  can  then  superimpose  different  texture  patterns  on  the 
original  image  with  each  single  element  of  the  texture  pattern  corresponding  to  a  single  cell.  We 
use  different  texture  elements/pattems  for  the  PD  (+)  and  PD  (-)  regions  of  the  image  [31]. 
Here  we  show  one  case  of  such  mapping.  We  have  generated  some  form  of  texture  pattern  by 
having  spatial  and/or  temporal  modulation  of  the  intensity  of  the  processed  PS  image  in  the 
regions  with  the  PD  (+)  and  PD  (-)  values.  Such  modulation  can  be  written  as 


/(+)  (x,y, n)  =  TPS  \_psl {x,y )]{l  +  M+  cos 
I{~]  (x,y,n)  =  Tps[PSI  (x,y)]\l  +  M-Cos 


[Q/b  +  /ex  sin 
+  /ex  sin 


(p)-JO'cos(p)]} 

L+yj-XTcos^ 


where  M+,  M  ,  Q 


were  defined  above,  k-IkIK  with  A  being  the  spatial  wavelength  of  spatial  modulation,  <p  is 

the  direction  of  ||  polarization  component,  and  the  angle  —  is  added  to  (p  for  modulation  in  the 

(-)  region  to  indicate  the  direction  of  1  polarization  component.  Spatial  as  well  as  temporal 
modulations  of  intensity  of  the  PS  image  provide  texture  patterns  that  can  be  easily  detected  by 
the  human  observer, 'resulting  in  segregation  of  the  PD  (+)  and  PD  (-)  regions.  Figure  14 
illustrates  the  static  case  of  such  textures  using  the  spatial  modulations  for  the  2-patch  target. 
(This  static  texture  corresponds  to  Q  =  0  in  the  above  equations.)  For  the  spatial  and  temporal 
modulations  we  have  made  movie  files,  which  are  available  upon  request.  One  can  easily  see  the 
presence  of  the  two  regions  with  PD  (+)  and  PD  (-)  values. 


More  detailed  description  of  our  algorithms  can  be  found  in  our  manuscript  is  attached  in  the 
appendix. 
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(A)  (B) 


Fig.  14.  Static  texture  using  spatial  modulation  of  the  intensity  of  the  processed  PS  image  of  the  2-patch  targeUo  represent  regions 
with  PD  (+)  and  PD  (— )  .  (This  PS  image  without  modulation  was  originally  taken  and  used  in  our  work  in  Hef  [26])  Panel  (A): 
M+-M_=  0.05  ,  K  =  1 ;  Panel  (B):  =  M_  =  0.25  ,  K  =  1 . 


6. Instrumentation  and  Camera  Systems  for  Polarimetric  Imagine 

During  our  efforts  in  this  period,  we  have  developed  various  camera  and  imaging  systems 
capable  of  collecting  images  with  the  polarization  information.  We  have  (a)  two  digital  cameras 
with  a  polarizing  beam  splitter,  capable  of  collecting  two  simultaneous  images  with  orthogonal 
polarizations  (Fig.  15  A);  (b)  a  digital  camera  (one  of  the  cameras  in  (a))  that  can  capture  images 
with  various  polarization  using  a  polarization  analyzer  in  front  of  it;  (c)  4-video-board-camera 
system  that  are  placed  on  a  platform  and  can  obtain  images  at  4  different  polarization  orientations 
(Fig.  15B);  (d)  individual  video  cameras  equipped  with  a  ferroelectric  liquid  crystal  rotator;  and 
(e)  a  six-camera  system;  two  in  the  visible  range,  two  in  the  near-infradred  (NIR);  and  two  in  the 
far-infrared  (FIR)  regimes  (Fig.  15C).  With  these  imaging  systems,  we  have  been  collecting 
useful  images  for  our  research  in  bio-inspired  polarimetric  sensing. 
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Fig.  15.  Some  of  the  imaging  systems  used  in  our  research  efforts. 
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1.  Introduction 

Polarization  is  an  important  feature  of  electromagnetic  waves,  and  it  can  be  affected  by  surface  shapes, 
materials,  local  curvature  and  features,  and  relative  location  of  sources  and  objects,  and  thus  it  can  provide 
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useful  information  about  the  observed  scene  and  objects.  Of  the  several  characteristics  of  visible  light,  only 
two  -  intensity  and  wavelength  -  are  received  and  encoded  by  the  human  eye  and  mapped  by  the  visual 
system  into  perceptual  qualities  of  brightness  and  color.  In  contrast,  without  appropriate  instruments 
human  eyes  cannot  effectively  utilize  light’s  polarization.  However,  it  is  well  known  that  eyes  of  some 
animal  species  are  sensitive  to  the  polarization  of  light  (see  e.g.,  [l]-[5]).  Polarization  sensing  by  some  of 
these  species  has  mostly  been  shown  to  provide  information  for  navigation  [5],  as  originally  discovered  by 
the  Nobel  laureate  Karl  von  Frisch  in  his  work  with  honeybees  [1],  and  it  has  been  suggested  that  some 
species  may  have  evolved  polarization  sensitivity  as  a  mechanism  of  enhancing  the  contrast  of  targets  in 
scattering  media  [6],  In  our  previous  work,  we  have  shown  that  an  optical  imaging  system  utilizing  such 
contrast  enhancement  through  polarization  differencing  can  increase  the  distance  over  which  targets  can  be 
detected,  and  their  critical  features  discriminated  [7]-[9],  Thus,  “polarization  imagery”  is  a  naturally 
occurring  and  demonstrably  successful  strategy  for  enhancing  vision. 

However,  since  the  human  eye  cannot  “see”  the  polarization  information,  when  this  information  is 
captured  by  a  polarimetric  imaging  system,  it  has  to  be  displayed  into  some  form  of  visual  cues/information 
that  can  be  detectable  by  a  human  observer.  In  other  words,  some  form  of  “sensory  substitution”  should  be 
exploited  for  representing  polarization  “signals”.  In  our  group’s  earlier  work,  we  have  shown  one  such  bio¬ 
inspired  mapping,  in  which  polarization  information  was  pseudo  color-coded,  based  on  the  opponent-colors 
model  of  human  vision  [9].  The  results  established  several  promising  imaging  strategies  for  displaying  in  a 
natural  way  the  contrast  enhancement  of  polarization  imagery  of  objects  in  scattering  media.  Such  mapping 
of  polarization  information  into  pseudo-colors  ignores  the  “true”  colors  of  the  scene,  namely,  the  spectral 
information  contained  in  the  scene.  So  if  one  wants  to  preserve  the  spectral  and  luminance  information  in 
an  image,  one  will  need  to  map  the  polarization  into  visual  cues  other  than  the  color  and  brightness. 

We  are  interested  to  explore  and  investigate  certain  bio-inspired  display  methodologies  for  mapping 
polarization  information  into  visual  information  that  can  be  readily  perceived  by  the  human  visual  system. 
In  this  Letter,  we  describe  one  such  mapping,  namely  representation  of  polarization  information  by  a  set  of 
coherently  moving  dots  superimposed  on  an  image.  It  is  known  that  human  vision  is  capable  of  motion 
perception,  which  includes  coherent  motion  detection,  form  from  motion,  and  biological  motion  (e.g.,  [10]- 
[12]).  We  exploit  the  sensitivity  of  coherence  detection  in  human  eye  in  displaying  polarization 
information  in  an  image.  One  of  the  motivations  behind  using  coherently  moving  dots  in  such  mapping  is 
to  preserve,  by  and  large,  various  features  of  the  image,  such  as  color  and  luminance,  while  “displaying” 
the  polarization  information. 


2.  Mapping  of  Polarization-Difference  (PD)  Signal  into  Coherently  Moving  Dots 

In  our  earlier  work,  we  introduced  the  concept  of  polarization-difference  imaging  [7].  In  this  technique, 
at  every  pixel  of  the  image,  the  intensities  of  the  two  orthogonal  polarization  components  of  the  light 
reaching  the  camera,  (jc,  >>)  and  /1(x,  y) ,  are  obtained,  and  then  the  sum  and  difference  of  these  two 
quantities  are  evaluated 

ps1(x>y)  =  I\i(x>y)+I±(x>y)  >  0) 

PDl(x,y)  =  Il](x,y)-Ix(x,y),  (2) 

where  (x,y)  identifies  the  pixel  position  in  the  image,  ||  and  1  indicate  two  orthogonal 

linear  polarizations,  PD  stands  for  the  “polarization-difference”  signal,  and  PS  for  the 
“polarization-sum”  signal.  If  an  ideal  linear  polarization  analyzer  is  used  to  capture 
orthogonal  polarization  components,  the  PS  image  will  be  equivalent  to  a  conventional 
intensity  image. 

In  order  to  use  moving  dots  to  represent  the  polarization  information  in  a  scene,  we  first  obtain  the  two 
orthogonal  polarization  components  at  every  pixel,  and  then  form  the  PS  image  of  the  scene  following  Eq. 
(1).  We  then  superimpose  dots  on  this  PS  image.  The  original  locations  of  dots  are  chosen  as  follows:  The 
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entire  PS  image  is  divided  into  small  identical  square-shape  cells  each  having  mxm  pixels,  e.g.,  10x10 
pixels.  In  each  cell,  a  single  square  dot  is  placed  at  a  random  location  within  the  cell.  The  size  of  the  dot  is 
nxn  pixels,  with  n«m,  and  can  be  properly  chosen  to  make  the  dots  small  enough  so  that  the  original 
image  would  be  practically  undistorted  by  the  presence  of  these  dots  and  large  enough  so  the  dots  can  be 
detected  by  an  observer.  In  the  results  presented  in  this  Letter,  the  dot  size  is  taken  to  be  2x2  pixels  for  an 
observer  comfortably  seated  in  front  of  a  computer  while  viewing  the  image  on  the  computer  monitor  at 
viewing  distance  of  about  50  cm.  So  in  each  cell,  the  2 x 2-pixel  dot  replaces  a  randomly  chosen  2x2- 
pixel  set  in  the  cell.  The  density  of  the  dot  depends  on  the  cell  density.  The  brightness  of  the  dot  is  chosen 
according  to  one  of  the  two  algorithms  described  later.  To  produce  dot  motion,  we  make  sequence  of 
several  frames  of  the  same  image,  but  in  each  frame  the  location  of  the  single  dot  in  each  cell  is  changed 
with  respect  to  its  location  in  the  previous  frame  as  follows.  (1)  if  the  averaged  PD  signal,  i.e.,  the  average 
of  PDl(x,y) ,  in  that  cell  is  greater  than  certain  “threshold”  value  8 ,  the  direction  of  change  of  location  of 

the  dot,  which  becomes  the  direction  of  perceived  motion  of  dot  in  that  cell  will  be  chosen  along  a  vector 
parallel  with  ||  polarization  component;  (2)  if  the  averaged  PD  signal  in  that  cell  is  less  than  —8 ,  the 
direction  of  dot  motion  will  be  selected  along  the  _L  polarization  component;  and  finally  (3)  if  the  averaged 
PD  value  is  between  -8  and  8  within  that  cell,  the  sequence  of  location  of  the  dot  in  that  cell  will  be 
picked  randomly,  leading  to  a  random  motion  of  dots  in  this  case  (or  the  dot  location  can  be  fixed  resulting 
in  dots  with  no  movement).  So  when  these  frames  are  played  sequentially,  the  dots  in  all  the  cells  with 
average  PD  signals  greater  than  8  appear  to  be  moving  coherently  in  the  direction  parallel  with  the  || 
polarization  component,  while  the  dots  in  cells  with  average  PD  signals  less  than  —8  move  coherently 
along  the  orthogonal  direction.  The  dots  in  cells  with  average  PD  signal  between  —8  and  8  move 
randomly  (or  they  do  not  move  at  all).  Since  the  human  visual  system  can  detect  and  perceive  coherent 
motion,  the  two  regions  with  dots  moving  coherently  in  two  orthogonal  directions  can  be  pre-attentively 
segregated  and  “popped  out”  against  the  rest  of  the  image  where  the  dots  are  moving  randomly  (or  they  are 
fixed).  Thus,  in  this  technique,  the  observer  views  the  PS  image  with  its  details  as  a  conventional  image, 
while  detecting  three  regions  in  the  image  representing  areas  with  PD  >  8 ,  PD  <  -8 ,  and  -8  <  PD  <  8 
signals. 

The  brightness/intensity  of  the  dots  must  be  chosen  such  that  they  are  easily  detectable  against  the 
background  image.  Here  we  suggest  two  different  recipes;  (1)  the  intensity  of  each  pixel  of  a  dot  in  each 
cell  is  selected  to  be  in  “contrast”  with  the  PS  signals  of  the  pixels  the  dot  is  replacing.  Specifically,  if 
PSl{x,y)  of  each  of  the  pixels  which  are  being  replaced  by  the  mxm-pixel  dot  is  greater  than  128  pixel 
intensity  in  an  8-bit  display  system,  the  intensity  of  each  pixel  of  the  dot  in  that  cell  will  be  assigned  as 
Ijoi{x>y)=  psl{x,y)~  128.  However,  if  psl(x,y)  <  128 ,  then  we  will  choose 

fthi{'x’'y)=  rsl{x:-y)'~  128-  Jn  this  scheme,  which  we. call  the  “contrast  scheme”,  the  dot  intensity  is 

“complemented”  against  the  background  intensity  in  each  cell;  (2)  in  this  method,  the  intensity  of  each 
pixel  in  a  dot  in  each  cell  is  chosen  to  be  A/%  less  than  the  PS  signal  in  the  image  pixel  which  it  is 
replacing.  So  Idol  {x,y)  =  (l- Af/100)  PSl{x,y) .  This  approach,  we  call  “percentage  scheme”. 


3.  Results 

To  demonstrate  this  mapping  strategy,  we  develop  the  above  algorithm  in  the  Matlab®  environment. 
The  frame  sequences  generated  using  this  algorithm  are  presented  as  movies  with  20  frames/second  in  the 
.avi  format.  These  movies  can  be  viewed  by  any  player,  e.g.,  Microsoft  Windows  Media™  Player,  Apple 
QuickTime™  Player,  or  the  likes. 

First,  in  a  dark  background  with  no  target,  we  show  how  a  region  with  coherently  moving  dots  can  be 
easily  segregated  against  the  rest  of  the  image  with  randomly  moving  dot.  Figure  1  shows  a  collection  of 
randomly  located  white  dots  on  a  dark  background.  When  one  clicks  on  the  link  to  the  movie  file  of  this 
Figure,  a  movie  appears,  and  one  can  see  that  dots  in  certain  rectangular  region  move  coherently  while  the 
dots  in  the  rest  of  the  image  move  randomly.  The  coherent  motion  of  dots  can  be  easily  detected,  and  the 
region  containing  these  dots  is  readily  noticed  by  the  observer. 
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We  now  apply  this  algorithm  to  the  images  of  a  target  that  was  previously  used  by  our  group  in  the  study 
of  polarization-difference  imaging  [7]-[9].  We  use  the  same  PS  and  PD  images  used  in  (Fig.  1  in  [9])  in 
order  to  allow  the  fair  comparison  between  mapping  of  polarization  using  the  new  algorithm  described  in 
this  Letter  with  that  of  the  colorimetric  PDI  algorithm  we  employed  in  [9].  The  target  was  an  aluminum 
disk  with  3.8-cm  diameter.  The  surface  of  this  disk,  except  for  two  1-cm2  square  patch  areas,  was 
sandblasted  to  make  it  Lambertian.  The  patches  were  raised  a  few  mills  and  were  abraded  with  the  emory 
paper  in  orthogonal  directions.  This  target  was  specifically  constructed  as  such  in  order  to  have  a  surface 
that  may  give  rise  to  scattered  light  with  partial  polarization  from  the  patch  areas  (each  with  partial 
polarization  parallel  with  the  direction  of  abrasion),  and  scattered  light  with  essentially  no  polarization  from 
the  sandblasted  surface.  Figure  2  shows  the  PS  and  PD  images  of  this  target  when  in  our  previous  study  it 
was  immersed  in  a  40-L  plexiglas  tank  of  water  to  which  5mL  of  milk  was  added,  and  it  was  front- 
illuminated  [9].  The  PD  image  was  scaled  using  a  “symmetric”  affine  transformation  [9]  to  utilize  the  full 
dynamic  range  of  the  8-bit  display.  In  other  words,  if  for  instance  the  range  of  PD  values  is  between  -a 
and  +b  (with  b>  a),  the  zero  value  of  PD  signal  is  mapped  to  pixel  intensity  128,  while  +b  and  -b  are 
mapped  to  255  and  0,  respectively.  These  images  were  originally  obtained  and  used  in  our  study  of 
colorimetric  PDI  by  Tyo,  Pugh,  and  Engheta  [9]. 

Figure  3  presents  implementation  of  the  current  mapping  scheme  on  the  image  of  our  target.  The 
threshold  value  is  taken  to  be  £  =  32  in  the  affine  transformed  PD  image  (which  is  effectively  a  threshold 
intensity  value  of  8  =  2  in  the  raw  PD  image  with  PD  value  range  between  ±8 .)  The  cell  size  is  10x10 
pixels  and  the  dot  size  is  2x2  pixels.  Dot  intensity  is  assigned  using  the  “contrast  scheme”  in  Fig.  3 A  and 
the  “percentage  scheme”  with  M-  30%  in  Fig.  3B.  (Click  on  appropriate  links  to  the  movie  file  of  each 
Figure  to  start  the  movie.)  Observing  these  movies,  one  is  able  to  distinguish  between  the  regions  with 
PD  >8  and  PD<-8  signals  (which  is  equivalently  PD  >8+ 128  and  .PD  <-£+128  in  the  affine 
transformed  PD  image),  and  the  region  with  -8  <PD<8  signal  (or  equivalently 
-£  +  128  <  PD  <£  +  128  in  the  affine  transformed  PD  image),  while  the  original  information  contents  of 
the  PS  image  is  essentially  intact.  With  regard  to  the  two  schemes  for  dot  intensity,  we  notice  that  the 
“contrast  scheme”  may  work  better  in  images  where  the  PS  intensity  distribution  may  be  somewhat 
uniform  over  certain  segment  of  the  image  (such  as  the  disk  face  in  our  example).  The  “percentage 
scheme”,  on  the  other  hand,  may  be  better  for  situations  in  which  unimportant  segments  of  an  image 
possess  the  PS  intensity  near  zero  (for  example,  the  dark  background  in  the  images  shown  here).  In  this 
way,  the  dots  in  such  regions  have  low  intensity,  and  thus  do  not  distract  the  observer  from  noticing  the 
dots  with  higher  intensity  in  regions  of  importance. 

Figure  4  shows  the  corresponding  results  when  the  threshold  value  is  chosen  to  be  8  =  48  in  the  affine 
transformed  PD  image  (i.e.,  effectively  threshold  value  of  3  in  the  raw  PD  image.)  (Click  on  appropriate 
links  to  the  movie  file  of  each  Figure  to  start  the  movies.)  Here  we  notice  that  smaller  regions  of  the  image 
possess  coherently  moving  dots,  which  represent  patch  areas  where  PD  signals  have  higher  absolute  values. 
So  the  choice  of  threshold  value  is  important  is  segregating  regions  of  the  image  with  specific  values  of  PD 
signals. 

Finally,  Fig.  5,  when  clicked  on  appropriate  links,  illustrates  a  similar  mapping  of  polarization  into 
moving  dots.  The  difference  here,  however,  is  that  the  dots  form  short  lines  with  time-varying  lengths  as 
they  move,  providing  additional  cues  for  perception  of  polarization  direction.  This  mapping,  of  course, 
results  in  higher  number  of  pixels  to  be  replaced  with  dots  and  lines,  but  it  may  provide  stronger  cues  for 
polarization  representation. 
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Fig.  I.  (2.5  MB)  A  collection  of  randomly  located  dots  on  a  dark  background.  Click  here  to  start  the  movie. 
One  can  see  that  the  region  with  coherently  moving  dots  can  be  easily  “popped  out”  against  the  background 
having  randomly  moving  dots.  Other  information:  image  size:  316x316  pixels,  cell  size:  7x7,  dot’s  pixel 
intensity:  255,  frame  rate:  20frames/sec,  dot’s  speed  in  the  region  with  coherently  moving  dots:  20  pixels/sec. 
(12  5  MB  version) 


Fig.  2.  (A)  PS  image  and  (B)  PD  image  of  the  target  to  which  we  apply  the  polarization-to-moving-dots 
mapping  strategy  introduced  here.  These  target  images  were  originally  obtained  and  used  in  our  previous  study 
of  polarization  difference  imaging  (PD1)  reported  in  (Fig.  1  in  [9]).  The  light  scattered  from  the  two  square 
patch  areas  are  slightly  partially  polarized  parallel  to  the  direction  of  abrasion  on  these  patches.  The  goal  here  is 
to  map  the  polarization  information  contained  in  the  PD  image  into  the  PS  image  by  using  coherently  moving 
dots.  Image  size:  512  x  479  pixels. 


IA)  (B) 

Fig.  3.  (Click  on  A  and  B  to  see  the  movies.)  Implementation  of  the  mapping  technique  described  here  on  the  image  of 
target  shown  in  Fig.  2..  .Polarization  information  from  the  affine  transformed  PD. image  (Fig.  2B)  is  mapped  as  moving 
dots  onto  the  PS  image  (Fig.  2A).  Here  the  threshold  value  is  chosen  to  be  8  =  32  for  the  affine  transformed  PD 
values.  In  (A),  the  dot  intensity  is  prescribed  using  the  “contrast  scheme”,  while  in  (B)  it  is  chosen  using  the 
“percentage  scheme”  with  M  =  30%.  Viewing  the  moving  dots,  our  visual  system  can  distinguish  among  the  regions 
with  PD  >  8  ,  PD  <  -8 ,  and  -8  <  PD  <  8  PD  signals.  Other  information:  Image  size:  340  x  316  pixels,  cell  size:  7x7 
pixels,  frame  rate:  20frames/sec.  (A- 1 4.9  MB  version,  B-10.3  MB  version). 


(A)  (2.54  MB)  IB)  (1.5 1MB) 
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Fig.  4.  (Click  on  A  and  B  to  see  the  movies.)  Similar  to  Fig.  3,  except  here  the  threshold  value  is  chosen  to  be  S  =  48 
for  the  affine  transformed  PD  values.  We  note  that  the  higher  threshold  value  results  in  having  smaller  regions  with 
coherently  moving  dots,  thus  highlighting  the  patch  areas  where  the  PD  signal  has  higher  absolute  values.  Other 
information:  Image  size:  340  x  316  pixels,  cell  size:  7x7  pixels,  frame  rate:  20  frames/sec.  (A-12.7  MB  version,  B-10.4 
MB  version). 


(B)  (1.2  MB) 

Fig.  5.  (Click  on  A  and  B  to  see  the  movies.)  Similar  description  as  Fig.  3,  except  here  the*moving  dots  form  short  line 
with  time-varying  lengths,  resulting  in  additional  cues  to  visualize  polarization  information  from  the  PD  image  given  in 
Fig.  2B.  Other  information:  Image  size:  340  x  316  pixels,  cell  size:  7x7  pixels,  frame  rate:  20  frames/sec.  (A- 12.9  MB 
version,  B-6.15  MB  version). 

The  above  examples  clearly  demonstrate  the  possibility  of  exploiting  the  sensitivity  of  human  visual 
systems  to  coherent  motion  as  one  possible  representational  mechanism  for  visualization  of  polarization 
information  in  an  image.  This  provides  the  observer  with  additional  information  about  the  scene  with  little 
or  no  distortion  to  the  other  non-polarization-related  information,  such  as  color  and  luminance,  in  the 
image.  This  mapping  represents  one  possible  “sensory  substitution”  for  displaying  polarization  information 
to  human  observers  while  retaining  the  conventional  image  information. 

Polarization  is  a  manifestation  of  the  vectorial  nature  of  optical  waves,  and  the  display  method 
employed  here  is  designed  to  visualize  the  polarization  information  in  an  image.  However,  it  is  important 
to  point  out  that  the  same  display  technique  can  be  used  to  represent  other  vector  quantities  in  images. 
Applied  to  optical  polarization,,  we  have  specifically  used  this  technique  to  map  the  “polarization- 
difference  (PD)”  signal,  since  it  was  shown  by  Tyo  [13]  that  the  PD  signal  is  indeed  uncorrelated  or 
“orthogonal”,  in  the  information  theoretic  sense,  to  the  “polarization-sum  (PS)”  signal,  i.e.,  to  the 
conventional  intensity  signals.  Thus,  ,  PD  and  PS  channels  constitute  together  an  optimum  basis  of  the 
polarization  components.  Since  we  want  to  superimpose  PD  information  on  a  PS  (i.e.,  intensity)  image,  we 
use  coherently  moving  dots  to  represent  the  PD  information  arising  in  optical  polarization.  In  this  context 
it  bears  emphasis  that  for  a  circularly-polarized  light,  the  PD  value  is  zero  and  thus  no  coherently  moving 
dots  are  associated  with  it.  Such  neglect  of  circular  polarization  is  justifiable,  since  in  any  optical  imaging 
system  in  which  the  phase  information  between  the  two  orthogonal  polarization  components  is  not 
available,  a  circularly-polarized  light  would  be  “equivalent”  to  an  unpolarized  light,  thus  resulting  in  a  zero 
PD  value.  Were  circular  polarization  information  available,  one  could  conceivably  employ  a  set  of  dots 
that  are  coherently  moving  in  a  circular  pattern  to  represent  the  local  circular  polarization.  This  requires 
further  investigation. 

5.  Conclusions 

We  have  introduced  a  visualization  technique  for  mapping  polarization  information  into  an  image 
without  essentially  altering  the  information  contents  of  the  original  image.  Dots  have  been  superimposed 
on  the  image,  and  the  motion  of  such  dots  has  been  implemented.  Regions  with  pd>8  and  PD<-8 
signals  are  given  dots  that  move  coherently  in  two  orthogonal  directions  representing  main  directions  of 
partial  polarization,  and  the  regions  with  -8<pd<8  signals  are  assigned  randomly  moving  dots.  Since 
human  vision  can  sense  and  perceive  coherent  motion,  segments  of  the  image  with  coherently  moving  dots 
can  be  readily  detected  and  distinguished  against  the  region  with  random  motion.  In  this  way,  the  segments 
with  various  ranges  of  PD  signals  “pop  out”.  Such  polarization  representation  can  lead  to  visibility 
enhancement,  better  target  detection  and  feature  extraction. 


(A)  (2.58  MB) 
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ABSTRACT 

For  imaging  systems  the  polarization  of  electromagnetic  waves  carries  much  potentially  useful  information 
about  such  features  of  the  world  as  the  surface  shape,  material  contents,  local  curvature  of  objects,  as  well 
as  about  the  relative  locations  of  the  source,  object  and  imaging  system.  The  imaging  system  of  the  human 
eye  however,  is  “polarization-blind”,  and  cannot  utilize  the  polarization  of  light  without  the  aid  of  an 
artificial,  polarization-sensitive  instrument.  Therefore,  polarization  information  captured  by  a  man-made 
polarimetric  imaging  system  must  be  displayed  to  a  human  observer  in  the  form  of  visual  cues  that  are 
naturally  processed  by  the  human  visual  system,  while  essentially  preserving  the  other  important  non¬ 
polarization  information  (such  as  spectral  and  intensity  information)  in  an  image.  In  other  words,  some 
forms  of  sensory  substitution  are  needed  for  representing  polarization  “signals”  without  affecting  other 
visual  information  such  as  color  and  brightness.  We  arc  investigating  several  bio-inspired  representational 
methodologies  for  mapping  polarization  information  into  visual  cues  readily  perceived  by  the  human  visual 
system,  and  determining  which  mappings  are  most  suitable  for  specific  applications  such  as  object 
detection,  navigation,  sensing,  scene  classifications,  and  surface  deformation.  The  visual  cues  and 
strategies  we  are  exploring  are  the  use  of  coherently  moving  dots  superimposed  on  image  to  represent 
various  range  of  polarization  signals,  overlaying  textures  with  spatial  and/or  temporal  signatures  to 
segregate  regions  of  image  with  differing  polarization,  modulating  luminance  and/or  color  contrast  of 
scenes  in  terms  of  certain  aspects  of  polarization  values,  and  fusing  polarization  images  into  intensity-only 
images.  In  this  talk,  we  will  present  samples  of  our  findings  in  this  area. 

Keywords:  imaging  systems,  display,  polarization  vision,  motion,  texture,- image  fusion,  temporal  fusion, 
flicker 


1.  INTRODUCTION 

1.1.  Motivation  and  objectives 

Polarization  is  an  important  feature  of  electromagnetic  waves,  and  can  be  affected  by  various  factors  in  the 
scene  such  as  surface  shapes,  surface  curvatures  and  materials,  and  position  of  objects  with  respect  to  the 
source(s)  and  the  observer,  and  therefore  it  can  provide  useful  information  about  the  observed  scene  and 
objects  -  information  which  otherwise  would  be  lost.  Two  characteristics  of  visible  light,  i.e.,  intensity  and 
wavelength,  can  be  detected  and  encoded  by  the  human  visual  system  and  interpreted  as  perceptual 
qualities  of  brightness  and  color.  Another  important  characteristics  of  light,  namely,  polarization,  however, 
cannot  be  effectively  exploited  by  unaided  human  eyes.  The  eyes  of  some  animal  species  (e.g.,  honeybees, 
desert  ants),  however,  can  sense  the  polarization  of  light  (see  e.g.,  ''°).  Such  polarization  vision  in  those 
species  has  often  been  shown  to  facilitate  their  navigation0.  Moreover,  it  has  been  suggested  that  some 
species  may  utilize  the  polarization  sensitivity  to  enhance  the  contrast  of  targets  in  scattering  media.6 


*engheta@.ee. upenn.edu:  phone:  +1-215-898-9777;  fax  +1-215-573-2068; 


Page  38 


Motivated  and  inspired  by  the  visual  systems  of  some  of  those  animal  species  with  polarization  vision  and 
particularly  by  the  concept  of  polarization  opponency  (analogous  to  color  opponency),  we  have  been 
investigating  novel  imaging  methodologies  based  on  the  concept  of  polarization  “difference”  signals.  In 
our  previous  work,  we  have  shown  that  an  optical  imaging  system  utilizing  contrast  enhancement  through 
polarization  differencing  can  increase  the  distance  over  which  targets  can  be  reliably  detected7'9,  and  have 
demonstrated  that  polarization-difference  imaging  (PDI)  is  indeed  a  useful  tool  for  enhancing  vision. 
However,  an  interesting  issue  arises  in  dealing  with  representing  the  polarization  information  in  a 
polarimetric  imaging  system.  Since  the  human  eye  cannot  directly  “see”  polarization  information,  how  can 
then  such  polarization  information  that  is  obtained  by  a  polarimetric  imaging  system  be  presented  to  a 
human  observer?  After  all,  the  outputs  of  any  imaging  system  must  be  presented  in  such  a  manner  that  can 
be  interpretable  and  recognizable  by  a  human  observer.  What  this  implies  is  that  such  information  should 
be  mapped  into  visual  cues  that  can  be  easily  detectable  by  a  human  observer,  hopefully  without  altering 
the  other  information  carried  by  cues  such  as  spectral  and  luminance  information.  In  our  group’s  earlier 
work,  we  have  shown  one  such  bio-inspired  mapping  for  the  cases  where  the  original  spectral  information 
(i.e.,  the  perceived  colors)  of  the  scenes  were  not  of  interest.  In  this  mapping,  polarization-difference 
information  was  pseudo-color-coded,  based  on  the  opponent-colors  model  of  human  vision9,  resulting  in  a 
contrast  enhancement  of  polarization  imagery  of  objects  in  scattering  media  such  the  objects  became  easily 
detectable  by  a  human  observer.  However,  that  mapping  ignored' the  spectral  infomiation  in  the  original 
scene,  information  normally  mapped  into  perceived  color.  In  order  to  preserve  the  spectral  and  luminance 
information  in  an  image,  we  need  to  map  the  polarization  into  visual  cues  other  than  the  color  and 
brightness. 

There  are  a  great  variety  of  visual  cues  that  are  detectable  by  the  human  visual  system.  These  include 
color,  intensity,  coherent  motion,  texture,  flicker,  to  name  a  few.  In  this  paper,  we  use  several  visual  cues 
as  well  as  certain  image  fusion  techniques  in  order  to  present  polarization  information  in  a  form  perceptible 
to  human  vision. 

1.2.  Review  of  the  Concept  of  Polarization-Difference  Imaging 

In  this  subsection,  we  give  a  brief  overview  of  the  Polarization  Difference  Imaging  (PDI)  technique.  This 
algorithm  was  first  introduced  in  Ref  [7]  and  was  further  enhanced  and  developed  in  Refs.  [8-9].  In  this 
technique,  the  intensities  of  the  two  orthogonal  polarization  components  of  the  light  detected  at  each  pixel 
( x,y )  of  the  image  are  obtained,  and  then  the  sum  and  the  difference  of  these  two  quantities,  I^(x,y)  and 

I±(x,y)  ,  are  evaluated.  These  sum  and  difference  signals  are  given  as  follows: 


J(x>y)=IAx>y)+Ii(x’y)> 

(3) 

■>I\x>y)  —  /„  {x,  y )  —  lL  (x,  k)  , 

(4) 

where  ||  and  1  indicate  parallel  and  perpendicular  components  of  the  polarizations  (the 
“parallel”  and  “perpendicular”  directions  are  with  respect  to  a  given  orientation  of  the 
polarization  analyzer),  PD  stands  for  the  “polarization-difference”  signal,  and  PS  for  the 
“polarization-sum”  signal.  Obviously,  for  an  ideal  linear  polarization  analyzer  the  PS 
image  will  be  equivalent  to  a  conventional  intensity  image. 

In  this  paper,  we  apply  our  visualization  techniques  to  several  different  man-made,  as 
well  as,  natural  targets.  The  first  man-made  target,  already  used  in  our  previous  study9, 0 
is  an  aluminum  disk  of  3.8-cm  diameter  with  its  surface  sandblasted,  except  for  two  1- 
cm2  square  regions  or  patches.  The  sandblasting  caused  the  surface  to  be  Lambertian. 
The  patches  were  raised  a  few  mills  and  were  abraded  with  emory  paper  in  orthogonal 
directions.  PS  and  PD  images  of  this  target,  when  front  illuminated,  are  presented  in 
Figs.  1A  and  IB,  respectively,  where  the  PD  image  intensities  were  scaled  using  a 
“symmetric”  affine  transformation9  to  utilize  the  full  dynamic  range  of  the  8-bit  display. 
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For  instance,  if  the  range  of  PD  values  is  between  -a  and  +b  (with  b  >  a  >  0  ),  the  zero 
value  of  PD  signal  is  mapped  to  pixel  intensity  128,  while  +b  and  -b  are  mapped  to  255 
and  0,  respectively.  (These  Figures  1A  and  IB  were  originally  obtained  and  used  in  our 
study  of  colorimetric  PDI.9)  The  second  specially  designed  target  was  another  aluminum 
disk  with  5.1 -cm  diameter  created  in  a  similar  way  as  in  the  first  one;  however,  this  disk 
has  seven  patches,  six  of  which  are  abraded  with  the  emory  paper  in  various  specific 
directions.  The  abrasions  (i.e.,  scratches)  on  these  six  patches  form  three  pairs.  Each  pair 
has  two  orthogonal  orientations  for  the  scratches.  More  specifically,  the  orientations  of 
the  scratches  on  these  six  patches  are  0",  30',  60“ ,  90",  120°,  and  150”  (with  respect  to  the 
vertical  axis.),  where  the  patch  pairs  with  orthogonal  orientations,  i.e.,  the  pairs  with 
(o  ,90") ,  (30°,  120“) ,  and  (60", 150")  are  located  diametrically  on  the  disk.  The  surface  of 

the  seventh  patch,  located  at  the  center  of  the  disk,  is  sandblasted  and  thus  is  Lambertian. 
The  affine  transformed  PS  and  PD  images  of  this  target,  in  the  back-illuminated 
condition,  are  presented  ip  Figs.  1C  and  ID,  respectively.  These  targets  were  specifically 
constructed  in  order  to  make  the  light  scattered  from  the  patch  areas  partially  polarized 
(with  the  major  axis  of  the  partial  polarization  parallel  to  the  direction  of  abrasion),  while 
the  light  scattered  from  the  sandblasted  areas  (including  the  7th  patch)  is  largely 
unpolarized.  (As  a  result,  in  the  PD  image  of  this  7-patch  target,  Fig.  ID,  the  centered  7th 
patch  is  indistinguishable  from  the  sandblasted  surface  of  the  disk,  and  thus  only  six 
abraded  patches  can  be  seen.)  For  detailed  description  of  the  experimental  setup,  see  Ref 
[8].  In  addition  to  these  specially  manufactured  targets,  we  also  use  here  some  natural 
targets  which  will  be  discussed  later. 

2.  DISPLAY  OF  POLARIZATION  INFORMATION  USING  COHERENTLY  MOVING 

DOTS 

The  first  visualization  scheme  we  present  here  is  the  representation  of  polarization  information  by  a  set  of 
coherently  moving  dots  superimposed  on  the  image.  We  exploit  the  sensitivity  of  the  human  eye  to 
coherent  motion  in  order  to  display  polarization  information  in  an  image.  It  is  known  that  human  visual 
system  is  capable  of  various  forms  of  motion  perception  (e.g.,  Refs  [11-14]).  One  of  the  motivations 
behind  using  coherently  moving  dots  in  such  mapping  is  to  preserve  the  original  non-moving  features  of 
the  image,  such  as  color  and  luminance,  while  “displaying”  the  polarization  information.  The  details  of  this 
technique  are  described  iri  our  recent  paper10.  Herevve  give  a  brief  review  of  this  technique. 

In  this  method,  many  dots  are  randomly  superimposed  on  the  entire  PS  image,  and  then  these  dots  are  made 
to  move  coherently  in  orthogonal  directions  according  to  the  value  of  the  PD  signal  at  the  location  of  the 
dots.  The  directions  of  dots  motion  are  chosen  in  the  following  way:  dots  in  the  region  with  associated  PD 
greater  than  a  certain  threshold  5  are  made  to  move  in  the  direction  parallel  with  ||  polarization 
component;  dots  in  the  regions  with  associated  PD  signal  less  than  —8  are  made  to  move  in  the  direction 
along  the  1  polarization  component;  dots  outside  the  above-mentioned  regions,  i.e.,  in  the  regions  with  PD 
value  between  -5  and  +S  are  made  to  move  randomly.  The  size  of  the  dot  is  properly  chosen  to  make 
the  dots  small  enough  so  that  the  original  image  would  be  practically  undistorted  by  the  presence  of  these 
dots  and  large  enough  so  that  the  dots  can  be  detected  by  an  observer.  The  brightness/intensity  of  the  dots 
must  be  chosen  such  that  they  are  easily  detectable  against  the  original  image.  Here  we  use  two  different 
methods: 

1)  The  intensity  of  each  pixel  of  a  dot  is  selected  to  be  in  “contrast”  with  the  PS  signals  of  the  pixels 
the  dot  is  replacing.  In  this  scheme,  which  we  call  the  “contrast  scheme”,  the  dot  intensity  is 
“complemented”  against  the  background  intensity  in  each  cell 

2)  In  the  second  method,  the  intensity  of  each  pixel  in  a  dot  is  chosen  to  be  M  percent  less  than  the 
PS  signal  in  the  image  pixel  which  it  is  replacing.  In  other  words, 
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hot  {x>y)  =  (1  -  A// 100)  psl(x,y) .  We  call  this  approach  “percentage  scheme”.  Detailed  analysis 
of  this  algorithm  can  be  found  in  Ref  [10]. 

The  representation  of  polarization  information  using  this  algorithm  was  implemented  in 
the  MATLAB®  environment.  To  illustrate  the  visualization  concept  of  coherently 
moving  dots,  we  use  PS  and  PD  images  of  the  2-patch  target  presented  in  Figs.  1A  and 
IB  (which  was  originally  given  in  Fig.  1  of  Ref  [9])  and  PS  and  PD  images  of  the  7-patch 
target  shown  in  Figs.  1C  and  ID.  We  have  developed  movie  files  for  these  images  with 
coherently  moving  dots  superimposed  (to  be  shown  in  the  presentation).  Here  in  this 
paper,  we  present  only  static  images  with  superimposed  dots  (i.e.,  a  single  frame  of  the 
movie)  corresponding  to  the  contrast  scheme  (Figs.  2A  and  2C)  and  percentage  scheme 
(Figs.  2B  and  2D).  In  Figs.  3A  and  3B,  we  also  show  the  same  images  but  with  “arrows” 
added  to  give  indication  of  the  direction  of  coherently  moving  dots.  (The  reader 
interested  in  receiving  the  movie  files  can  send  e-mail  to:  engheta@ee.upenn.edu.  The 
movie  files  of  moving  dots  on  the  2-patch  target  can  also  be  found  in  Ref  [10],  which  is 
published  in  the  online  journal  Optics  Express  with  multimedia  files  included.)  For  the 
percentage  scheme,  we  have  chosen  parameter  M  =  30%  .  When  these  movies  are  viewed, 
the  human  observer  can  easily  and  pre-attentively  segregate  the  region  containing 
coherently  moving  dots  from  the  regions  with  randomly  moving  dots.  The  significance  of 
this  result  is  that  polarization-difference  information  has  been  superimposed  on  the  image 
in  a  manner  that  allows  visualization  of  target  regions  having  similar  polarization 
signatures,  without  the  loss  of  the  luminance  information. 


Figure  1 :  PS  images  of  the  2-patch  target  front  illuminated  (A)  and  the  7-patch  target  back  illuminated  (C).  Panels 
(B)  and  (D)  are  the  PD  images  of  the  same  targets,  respectively,  to  which  we  apply  the  polarization-to-moving-dots 
mapping  strategy.  Image  (B)  has  been  “symmetrically”  affine  transformed9,  while  image  (D)  is  simply  affine 
transformed.  Images  (A),  (B)  were  originally  obtained  and  used  in  our  previous  study  of  polarization  difference  imaging 
(PDI)  reported  in  Ref  [9]  (Fig.  1  in  Ref  [9]). 
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Figure  2:  Panels  (A)  and  (C):  Superimposing  dots  on  the  PS  images  of  the  2-patch  and  7-patch  targets  using  the 
contrast  scheme.  Panels  (B)  and  (D):  corresponding  images  with  the  percentage  scheme.  Dot  size  is  lxl  pixel  in  Panels 
(A)  and  (B),  and  2x2  pixels  in  Panels  (C)  and  (D).  Images  in  Panels  (A)  and  (B)  were  originally  presented  in  Ref  [1 0] 


Figure  3:  The  same  as  in  Fig.  2,  except  that  some  “arrows”  have  been  added  to  give  indication  of  the  direction  of 
motion  of  the  coherently  moving  dots.  We  have  made  movie  files  showing  the  coherent  motion  of  these  dots.  Movies 
will  be  shown  in  the  presentation.  Interest  readers  can  contact  the  corresponding  author  to  get  copies  of  the  movie  files. 
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3.  REPRESENTATION  OF  POLARIZATION  INFORMATION  USING  TEMPORAL 

FUSION  TECHNIQUE 


A  variety  of  image  fusion  techniques  have  been  used  by  many  researchers  to  combine 
two  or  more  images  from  different  modalities  (such  as  images  from  the  visible  band  and 
the  IR  band)  into  a  single  image  that  ideally  contains  all  the  important  information  of 
original  images.15  In  principle,  those  fusion  techniques  can  also  be  used  to  fuse 
polarization  images  with  non-polarization  images  (e.g.,  intensity-only  images).  Our 
colorimetric  polarization-difference  imaging  technique,  developed  earlier,  is  indeed  a 
fusion  technique  in  representing  polarization  information  with  a  pseudo-color  mapping.9 
Here,  we  introduce  the  concept  of  “temporal  fusion”,  where  the  fusing  of  two  different 
images  is  achieved  in  time  domain  by  taking  advantage  of  the  biological  sensitivity  of 
human  visual  system  to  the  "fading  jn  and  fading  out"  between  two  image  modalities.  As 
applied  to  polarization,  the  image  modalities  of  interest  here  are  an  intensity-only  image 
(i.e.,  PS  image)  and  the  polarization-difference  (PD)  information  of  the  same  scene  (e.g., 
this  latter  image  could  be  a  grayscale  representation  of  the  PD  data,  or  a  pseudo-color 
representation).  A  dynamic  image  (i.e.,  a  movie)  consisting  of  a  gradual  periodic 
transition  from  a  PS  image  to  PD  image  is  then  created,  with  the  frequency  of  transition 
chosen  such  that  the  human  eye  can  easily  and  most  sensitively  detect  this  fading  in  and 
fading  out.  In  this  way,  the  observer  can  clearly  see  and  distinguish  the  PS  information 
and  PD  information  in  the  same  dynamic  image.  In  other  words,  we  take  advantage  of 
the  dimension  of  “time”  in  order  to  fuse  information  from  two  different  modalities,  i.e., 
from  polarization  and  non-polarization  information.  If  the  PD  image  is  formed  as  a 
pseudo-colored  image,  as  will  be  discussed  below,  then  we  choose  the  frequency  of 
fading  between  the  PS  and  PD  image  in  accordance  with  sensitivity  of  color  channels  in 
the  human  eye. 


In  our  temporal  fusion  technique,  one  image  (e.g.,  PS  image)  “fades”  into  the  other  (e.g., 
PD  image)  via  a  sequence  of  intermediate  image  frames,  each  of  which  combines 
information  from  PS  and  PD  images  with  different  weights.  The  period  of  one  complete 
cycle  is  selected  in  accordance  with  the  sensitivity  of  the  human  visual  system  to  flicker 
(see  e.g.,  Ref  [18]).  Our  algorithm  can  be  described  in  the  following  way: 

The  first  and  the  last  frames,  i.e.,  frame  numbers  n  =  1  and  n  =  N ,  of  each  complete  cycle 
of  the  movie  are  chosen  as  PS  images.  The  “intermediate  frames”  are  computed  pixel-by- 
pixel  from  PS  and  PD  images  using  the  following  expression: 

I ( x,y,n )  =  a(n)  TPD  [TO/(x,y)]  +  [l  -a{n)]TK  [PSI (jc.y)] , 

where  a(n)  is  a  temporal  fusion  weighting  function  of  the  frame  number  1  <n<N  .  The  value  of  a(n) 
varies  between  0  and  1  with  etr(l)  =  a(N)  =  0 ,  (x,y)  represents  the  pixel  location  in  the  image,  and  the 
operators  TPD  [  •  ]  and  Tps  [  •  ]  represent  transformation  (or  processing)  of  the  “raw”  PD  and  PS  data  into 

some  forms  for  PD  and  PS  image  representation.  For  instance,  these  operations  may  include  symmetric 
affine  transformation,  conventional  affine  transformation,  pseudo-colorimetric  (R,G,B)  or  (H,S,V) 
mapping,  to  name  a  few.  According  to  Eq.  (3),  every  frame  in  the  cycle,  between  n  =  1  and  n  =  N ,  is 


(5) 
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created  with  certain  percentages  of  PD  and  PS  information.  There  can  be  various  choices  for  the  function 
a(n ) ,  each  producing  different  visual  effects.  Here,  we  consider  two  cases  for  a(n) ,  viz.,  the  “Gaussian” 
and  the  “sinusoidal”  modulation.  For  the  Gaussian  modulation,  the  function  a(n)  is  set  to  be 
£*■(«)  =  exp (-(H-K0)J/2er)  for  [<n<N  where  n  is  current  frame  number,  na  is  the  frame  number  at 
which  only  PD  image  is  shown  (e.g.,  no  =  N / 2),  a2  is  the  variance  and  a(  1)  =  a(N)  =  0 .  For  sinusoidal 
modulation,  the  function  a(n)  is  considered  as  cc{n)  =  sin[;r(«-l)/(W-l)] .  The  frame  sequences 

generated  using  either  of  these  two  algorithms  have  30  frames  in  each  cycle,  and  are  compiled  into  movie 
files  with  playback  rate  of  30  frame/sec.  We  show  the  snapshots  of  these  movies  in  the  following 
subsections.  (Interested  reader  may  contact  the  corresponding  author  to  receive  these  movie  files.) 

3.1.  Temporal  fusion  of  grayscale  PS  and  PD  images 

In  this  subsection,  we  show  the  fusion  of  the  PS  and  PD  images  of  the  7-patch  target  as  8-bit  grayscale 
images.  The  snapshots  of  the  movies  which  illustrate  temporal  fusion  using  Gaussian  and  sinusoidal 
modulations  are  presented  in  Fig.  4  and  Fig.  5,  respectively.  As  can  be  seen  from  these  snapshots  (and  even 
more  clearly  from  the  movie  files),  wc  can  observe  both  the  polarization-information  (from  the  PD  image) 
and  the  intensity-only  information  (from  the  PS  image)  in  the  same  set  of  dynamic  images. 

3.2.  Temporal  fusion  of  grayscale  PS  and  pseudo-colorimetric  PD  images 

In  our  previous  work9,  we  developed  a  technique  for  mapping  polarization  information  into  pseudo-colors 
using  the  opponent  hues  consistent  with  the  opponent-colors  model  of  human  vision.  It  is  known  that  most 
colors  perceptible  by  the  human  eye  can  be  described  in  a  3-dimensional  color  space  using  three 
independent  parameters,  such  as  “brightness”,  “saturation”,  and  “hue”  in  the  HSV  (Hue-Saturation-Value) 
color  space.  For  the  pseudo-colorimetric  representation  of  the  PD  image  we  have  considered  several 
mapping  schemes.  For  our  temporal  fusion  technique  between  the  grayscale  PS  image  and  the  pseudo- 
colorimetric  PD  image,  we  use  the  following  mapping  for  the  PD  image,  as  done  in  Ref  [9]:  The  affine 
transformed  value  of  PS  image  intensity  at  any  pixel  is  mapped  into  the  “value”  (V)  of  the  HSV  map.  The 
sign  of  PD  signal  at  each  pixel  is  mapped  into  a  pair  of  opponent  hues,  i.e.,  positive  PD  is  mapped  into  one 
hue,  and  negative  PD  is  mapped  into  the  corresponding  opponent  hue.  The  magnitude  of  the  PD  signal  at 
each  pixel,  after  it  is  affine  transformed,  is  mapped  into  “saturation”  (S). 


Figure  4:  Snapshots  of  temporal  fusion  of  the  grayscale  PS  and  PD  images  of  the  back-illuminated  7-patch  target 
using  the  “Gaussian”  modulation  with  <T  =  2  .  Every  complete  cycle  of  the  movie  file  includes  30  frames,  which  can 
be  viewed  with  the  rate  of  30  frame/sec.  In  this  figure,  we  only  present  frames  No  1,  5,  10,  12,  15,  18,  25,  and  30,  from 
the  left  to  right  and  the  top  to  bottom. 
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frame  5 


frame  10 


frame  30 


frame  15 


frame  25 


frame 


Figure  5:  Similar  to  Fig.  4,  except  the  “sinusoidal”  modulation  is  used  in  the  temporal  fusion. 


In  Figs.  6  and  7  we  show  the  temporal  fusion  of  the  pseudo-colorimetric  representation  of  the  PD  image 
with  the  grayscale  PS  image  of  the  7-patch  target.  The  equations  used  here  are  similar  to  Eq.  (3)  as  in 
subsection  3.1  to  fuse  pseudo-colorimetric  PD  and  grayscale  PS  images.  Snapshots  of  the  resulting  movies 
are  given  in  Fig.  6  (with  Gaussian  variation  for  a(n)  )  and  Fig.  7  (with  the  sinusoidal  variation).  Here  the 
observer  can  get  the  information  from  the  grayscale  intensity-only  image  along  with  the  polarization 
information  from  the  pseudo-color  PD  image  in  the  same  set  of  dynamic  displays. 


frame  1 


frame  15 


frame  18 


frame  30 


frame  25 


Figure  6:  Snapshots  of  temporal  fusion  of  the  grayscale  PS  with  the  pseudo-colorimetric  PD  images  of  the  back- 
illuminated  7-patch  target  using  the  “Gaussian”  variation.  As  in  Figs.  4  and  5,  every  complete  cycle  of  the  movie  file 
includes  30  frames,  which  can  be  viewed  with  the  rate  of  30  frame/sec.  (The  movie  files  can  be  obtained  from  the 
corresponding  author) 
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frame  1 


frame  5 


frame  10 


frame  12 


frame  15 


frame  18- 


frame  25 


Figure  7 :  Same  as  Fig.  6,  except  the  “sinusoidal”  variation  is  used. 


3.3.  Temporal  fusion  of  a  color  image  with  its  pseudo-color  polarimetric  image 

In  this  subsection  we  consider  the  temporal  fusion  of  a  color  image  of  a  natural  scene  with  its  pseudo-color 
polarimetric  image.  It  is  well  known  that  for  the  light  coming  from  every  point  in  a  scene,  three  parameters 
related  to  the  polarization  information  detectable  by  a  2-D  imaging  sensor,  in  which  the  phase  information 
between  the  two  orthogonal  polarization  components  is  not  available,  can  be  described  as  the  energy 
intensity  (or  brightness)  1 ,  degree  of  linear  polarization  p ,  and  orientation  angle  (p0  of  the  polarization 
ellipse  of  the  polarized  portion  of  the  light  from  that  point.  The  value  of  p  varies  between  0  and  1  where 
p  -  0  presents  unpolarized  light  and  p  =  1  corresponds  to  a  completely  polarized  signal.  To  find  these 
parameters  at  every  pixel,  one  needs  to  collect  a  minimum  of  three  images  of  the  same  scene;  each  image 
taken  with  a  different  orientation  of  the  polarization  analyzer  in  front  of  the  camera,  e.g.,  with  0" ,  45” ,  and 
90"  orientation  of  the  polarization  analyzer.  These  three  independent  “measurements”  can  provide  one 
with  three  equations  from  which  the  parameters  I ,  p,  and  <po  can  be  evaluated  for  each  pixel.  To  display 
these  three  parameters  of  a  scene  in  a  single  image,  one  can  use  a  variety  of  mapping  strategies.  We  have 
investigated  several  different  mapping  possibilities.  One  such  mapping  is  the  Bernard  and  Wehner 
mapping.16  Bernard  and  Wehner  suggested  the  mathematical  similarities  between  a  theoretical  3-parameter 
polarization  vision  system  and  the  widely  used  3-parameter  color  vision  system.  They  also  pointed  out  that 
some  species  may  indeed  have  3-parameter  polarization  vision  system.16  The  main  analogy  they  draw  is  the 
parallelism  of  the  intensity  /  with  the  value  V,  the  degree  of  polarization  p  with  the  saturation  S,  and  the 

angle  (po  with  the  hue  H  (in  the  HSV  color  space).  This  parallelism  has  been  used  in  the  past  in  certain 
polarization-sensitive  imaging  systems.17  Here,  as  an  input  to  our  temporal  fusion  algorithm,  we  utilize  this 
pseudo-color  mapping  based  on  this  parallelism  to  produce  a  pseudo-color  polarimetric  image.  For  display 
purpose,  we  incorporate  the  image  enhancement  technique  “histogram  equalization”  on  the  original  value 
V  (in  the  HSV  space)  of  the  “true-color”  image,  and  use  this  equalized  V  as  the  V  channel,  and  the  contrast 
enhanced  p  data  for  the  saturation  channel  S.  For  the  angle  of  polarization  <po ,  we  map  2 (p0  to  the  hue 
angle.  As  an  example,  we  use  a  picture  of  flowers  and  leaves  which  we  have  taken  outdoors  by  taking 
three  images  with  three  orientations  of  the  polarization  analyzers  as  mentioned  above.  A  processed  “true- 
color”  image  of  this  scene  is  shown  in  Fig.  8A  after  histogram  equalization  has  been  applied;  the 
distribution  of  degree  of  linear  polarization  p(x,y) ,  after  contrast  enhancement,  is  shown  in  Fig.  8B;  and 
the  distribution  of  the  angle  of  polarization  ellipse,  <po(x,y),  is  depicted  in  Fig.  8C  with  the  angle-to-hue 
color  wheel  shown  at  its  right.  Figure  8D  illustrates  the  mapping  of  {l,p,(po)  into  ( V,S,H ) .  We  notice 
that  the  images  with  polarization  information  (Figs.  8B,  8C,  and  8D)  reveal  certain  additional  information 
not  available  in  the  regular  color  image  in  Fig.  8  A.  For  instance,  we  note  that  in  the  distribution  of  p{x,y) 
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shown  in  Fig.  8B  the  leaves  in  general  exhibit  higher  degree  of  linear  polarization  than  the  flowers  (the 
flowers  look  darker  than  the  leaves  in  p(x,y ) ).  Moreover,  we  observe  that  certain  parts  of  the  scene  with 
low  intensity  shown  in  the  conventional  image  in  Fig.  8A,  such  as  the  little  leaf  in  the  foreground  slightly 
below  the  three  large  leaves,  may  appear  more  noticeable  in  the  images  of  p(x,y)  and  <po(x,y)  shown  in 
Figs.  8B  and  8C,  since  the  degree  of  polarization  at  a  pixel  may  in  general  vary  independently  of  intensity 
of  that  pixel. 

We  have  applied  our  temporal  fusion  technique  to  fuse  image  shown  in  Fig.  8A  with  that  in  Fig.  8D  and  to 
produce  movie  files  for  such  fusion.  Figure  9  illustrates  the  snapshots  of  this  temporal  fusion  using  the 
sinusoidal  function  for  a(n) . 

4.  VISUALIZATION  OF  POLARIZATION  INFORMATION  BY  TEMPORAL  FLICKER 

In  this  method  of  presenting  polarization  information,  we  exploit  the  sensitivity  of  human  eye  to  intensity 
flicker  in  an  image17.  After  forming  the  PS  and  PD  images  of  a  scene,  we  intend  to  bring  the  observer’s 
attention  to  the  regions  of  the  image  having  PD  signals  with  opposite  signs.  We  find  these  regions  by 
searching  for  pixels  whose  raw.PD  values  are  greater  than  =?S  and.  those  that  are  less  than  —S ,  where  S  is 
a  given  threshold.  These  regions  we  name  (+)  and  (-)  regions,  respectively.  The  rest  of  the  image,  in 
which  —S  <  PD  <  S ,  is  the  “PD  neutral”  region.  Then  we  temporally  modulate  the  intensity  of  the  pixels 
in  PS  image  belonging  to  the  (+)  and  (-)  regions  from  frame  to  frame,  forming  a  movie  file.  Such 
temporal  modulation  can  be  expressed  as: 

7W  (x,y,n)  =  Tps  [„/(x,j/)](l  +  M+  sin  (Or,,))  (4) 

/H(x,y,«)  =  rra[w/(x,y)](l  +  M_sin(Q/„+^))  (5) 

Here,  x  and  y  are  coordinates  of  the  pixels  within  the  image,  n  is  the  frame  number  ranging  from  1  to  N 
where  N  is  the  number  of  frames  in  a  complete  cycle,  tn  corresponds  to  the  time  of  frame  n ,  M+  and 
are  the  modulation  coefficients  for  the  (+)  and  (-)  regions,  respectively,  £2  =  2k F  where  F  is  the 
frequency  of  flicker  in  Hz.  There  is  180"  phase  difference  between  the  flickers  in  the  (+)  and  (-)  regions. 

In  the  following  example,  this  visualization  technique  is  illustrated  with  M+  =  M  =  25%  and  N  =  30 . 

The  flicker  frequency  is  selected  to  be  F  =  1 ,  10,  and  15  Hz.  The  movies  will  be  demonstrated  in  the 
presentation.  The  snapshots  of  the  flicker  for  F  =  1  Hz  are  shown  here  in  Fig.  1 0. 
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Figure  8:  (A)  A  processed  “true  color”  image  of  a  scene  after  the  histogram  equalization  of  its  V  channel;  (B)  Degree 
of  linear  polarization  p(x,  _y)  image  after  it  is  contrast  enhanced;  (C)  Angle  of  polarization  ellipse,  (p0  ( X ,  y )  ,  shown 
in  pseudo-color  scheme.  The  angle-to-hue  color  wheel  is  shown  to  the  right;  and  (D)  Pseudo-color  mapping  of  three 
parameter  {I,p,<P0)  into  ( V,S,H )  . 


Figure  9:  Snapshots  of  temporal  fusion  of  Fig.  8A  (color  image  of  the  scene)  with  Fig.  8D  (pseudo-color  polarimetric 
image  of  the  same  scene)  using  the  sinusoidal  function  Cc(ri)  ,  Every  complete  cycle  of  the  movie  file  includes  30 
frames,  played  back  at  the  rate  of  30  frame/sec. 
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Figure  10:  Snapshots  of  flickering  technique  as  a  method  to  visualize  regions  with  “opponent  PD  signals”.  Here 
M+  =  M  —  25%  ,  N  =  30 ,  and  frames  1,  3,  5,  7,  11,  15,  18,  24,  27,  and  30.  (The  movie  files  can  be  obtained 
from  the  corresponding  author.) 


5.  USE  OF  STATIC  AND  DYNAMIC  TEXTURES  TO  DISPLAY  POLARIZATION 

INFORMATION 


In  this  subsection,  we  display  polarization  information  using  another  visual  cue,  namely,  texture.  It  is 
known  that  certain  forms  of  textures  can  be  segregated  by  the  human  visual  perception  (see  e.g.,  Ref  [19- 
21]).  We  have  been  exploring  how  some  textures  can  be  used  to  visualize  the  polarization  information. 


In  order  to  utilize  textures  for  this  purpose,  we  first  subdivide  the  entire  image  into  small  cells  of  mxm 
pixels  (e.g.,  m  =  10),  and  we  evaluate  the  average  PS  and  PD  values  in  each  cell.  We  then  identify  the  (+) 
and  (-)  regions  of  the  image  which  correspond  to  pixels  with  ( PD)ceU  >  +5  and  (PD)ccll  <  -8 ,  as  was 
described  in  Section  4.  Here  (PD)ccll  denotes  the  average  value  of  PD  in  each  cell.  We  can  then 
superimpose  different  texture  patterns  on  the  original  image  with  each  single  element  of  the  texture  pattern 
corresponding  to  a  single  cell.  We  use  different  texture  elements/pattems  for  the  PD  (+)  and  PD  (-) 
regions  of  the  image.  For  instance,  texture  patterns  with  circles  "°"  and  asterisks  can  be  used  for  the 
(+)  and  (-)  regions,  respectively.  Cells  with  -8<(PD)ccll  <8  have  no  texture  patterns.  Fig.  11A  shows 
a  PS  image  of  a  leaf  we  took  under  indoor  fluorescent  lighting  (this  image  is  histogram  equalized).  Image 
in  Fig.  1  IB  presents  the  superimposed  texture  patterns  for  threshold  8  =  0.03  (with  respect  to  the  raw  PD 
data  between  -1  and  +1 .  One  can  recognize  these  two  texture  patterns,  which  indicates  that  the  surface  of 
the  leaf  has  positive  PD  values.  (Here  the  ||  direction  is  horizontal.) 

Another  form  of  texture  pattern  can  be  generated  by  having  spatial  and/or  temporal  modulation  of  the 
intensity  of  the  processed  PS  image  in  the  regions  with  the  PD  (+)  and  PD  (-)  values.  Such  modulation 
can  be  written  as 


7(+)  ( x,y,n )  =  TPS[PSI (*,y)]{l  +  M+  cos[Qr„  +  rx  sin  ((p)-x'j' cos  ($?)]} 


/H  (x,y,n)  =  Tps  [  ra/(x,y)]|  1  +  M_  cos 


n 


Qtn  +o-sin|  <p  +  —  l-arycos 


f 

rij 


where  M+,  M  ,  Q  were  defined  in  Section  4,  k  =  2n  I  h  with  A  being  the  spatial  wavelength  of  spatial 

K 

modulation,  (p  is  the  direction  of  ||  polarization  component,  and  the  angle  —  is  added  to  <p  for 

modulation  in  the  (-)  region  to  indicate  the  direction  of  1  polarization  component.  Spatial  as  well  as 
temporal  modulations  of  intensity  of  the  PS  image  provide  texture  patterns  that  can  be  easily  detected  by 
the  human  observer,  resulting  in  segregation  of  the  PD  (+)  and  PD  (-)  regions.  Figure  12  illustrates  the 
static  case  of  such  textures  using  the  spatial  modulations  for  the  2-patch  target.  (This  static  texture 
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corresponds  to  ft  =  0  in  Eqs.  (6)  and  (7).)  For  the  spatial  and  temporal  modulations  we  have  made  movie 
files,  which  can  be  obtained  from  the  corresponding  author.  One  can  easily  see  the  presence  of  the  two 
regions  with  PD  (+)  and  PD  (-)  values. 


Figure  1 1 :  Using  textures  to  display  polarization  information.  Here  the  texture  patterns  with  circles  "  and  asterisks 
are  superimposed  in  the  PD  (+)  and  PD  (— )  regions.  (A)  Processed  PS  image  (with  histogram  equalization),  (B) 
superimposed  textures  with  8  =  0.03  . 


.(A)  ...  .  .  _  ......  ..  ...  ...  (B) 


Figure  12:  Static  texture  using  spatial  modulation  of  the  intensity  of  the  processed  PS  image  of  the  2-patch  target  to 
represent  regions  with  PD  (+)  and  PD  (— )  .  (This  PS  image  without  modulation  was  originally  taken  and  used  in  our 

work  in  Ref  [9])  Panel  (A):  M+  =  M_  -  0.05  ,  K  =  1 ;  Panel  (B):  M+  =  M_  =  0.25  ,  K  =  1 . 


CONCLUSIONS 

In  this  paper  we  have  discussed  several  different  visualization  techniques  for  representing  polarization 
information  using  visual  cues  detectable  and  recognizable  by  the  polarization-blind  human  eyes.  These 
include  using  coherently  moving  dots,  temporal  fusion  and  flickering,  and  texture  patterns  and  static  and 
dynamic  textures  by  spatial  and/or  temporal  modulations.  These  display  techniques  for  visualizing 
polarization  information  in  an  image  provide  the  segments  of  image  with  various  ranges  of  PD  signals  to  be 
segregated  and  “pop  out”  and  thus  allow  the  polarization-blind  human  observers  to  easily  “see” 
polarization  information.  Such  polarization  representation  can  provide  visibility  enhancement,  better  target 
detection  and  feature  extraction,  without  loss  of  the  information  that  is  present  in  the  image  that  would  be 
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collected  by  a  system  not  sensitive  to  polarization. 
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Introduction 

Fingerprinting  is  one  of  the  most  widely  used  biometric  methods  for  identifying  and 
authenticating  individual  persons.  The  modem  science  of  fingerprinting  started  in  the 
second  half  of  the  19th  century.  For  an  interesting  historical  review  see  references1'3. 
There  are  two  types  of  fingerprint  data,  distinguished  by  their  formation  processes.  In 
forensic  science  finger  marks  left  unintentionally  at  a  crime  scene  are  referred  to  as 
“latent  fingerprints”.  Fingerprints  acquired  directly  from  human  fingers  using  ink  or 
scanners  in  controlled  environments  are  referred  to  as  “exemplar  fingerprints” M. 
Although  both  types  of  fingerprints  are  related  to  some  extent,  the  recovery  of  “latent” 
and  “exemplar”  fingerprints  poses  very  different  technical  challenges.  There  have  been 
considerably  more  successful  optical  methods  for  exemplar  fingerprints  than  for  latent 
fingerprints  and  sometimes  the  classification  can  be  confusing.  For  example,  there  exist 
methods  using  “laser”  and  “polarization”  for  extracting  fingerprint  images  directly  from 
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live  human  fingers5;  such  methods  are  for  “exemplar  fingerprints”.  The  purpose  and  the 
detailed  physical  background  of  that  system  are  both  different  from  those  involved  in  the 
application  of  “laser”  or  “polarization”  to  “latent  fingerprints”.  Another  example  is  a 
device  that  claims  to  “optically  reads  a  latent  fingerprint”6.  However,  the  main  function 
of  the  particular  device  is  to  read  directly  from  a  human  finger  (and  simultaneously 
comparing  with  a  known  fingerprint  pattern)  and  such  device  should  be  classified  as  an 
“exemplar  fingerprint”  reader.  Recovery  of  latent  fingerprints  are  much  more  difficult 
than  the  recovery  of  exemplar  fingerprints  because  the  physical  and  chemical 
composition  of  latent  fingerprints  and  the  surfaces  on  which  they  are  found  vary  greatly 
and  can  often  undergone  unknown  degradation  before  being  examined.  In  this  paper  we 
present  a  new  method  for  the  detection  and  recovery  of  “latent  fingerprints”. 

Latent  fingerprints  differ  from  exemplar  fingerprints  in  that  they  are  very  difficult 
to  detect  with  unaided  human  vision  under  most  ordinary  viewing  conditions  (hence  their 
name);  they  are  usually  also  of  lower  quality  than  the  exemplar  fingerprints,  although 
high  quality  fingerprint  marks  can  at  times -also  be  found  at  a  crime  scene.  To  be  precise, 
non-exemplar  fingerprints  that  can  be  easily  seen  by  a  human  observer  should  be  called 
“patent  fingerprints”  M.  In  practice,  however,  the  term  “latent  fingerprints”  is  often  used 
to  refer  to  all  fingerprints  that  are  not  “exemplar”.  It  is  the  really  “latent”  fingerprints  that 
are  more  common  at  a  crime  scene  and  require  greater  efforts  to  render  visible.  Most 
techniques  employed  for  this  purpose  utilize  a  chemical  or  physical  process  that  applies 
some  kind  of  material  directly  to  the  surface  suspected  to  bear  fingerprints1'4’7'9.  Once  the 
contrast  of  the  fingerprint  mark  is  sufficiently  enhanced  by  such  treatments,  the  mark  is 
either  photographed  or  “lifted”  in  order  to  be  permanently  archived  as  evidence.  The  term 
“lifting  the  fingerprint”  originates  from  the  oldest,  but  still  widely  used  fingerprint 
detection  method  —  powdering  —  in  which  the  powders  applied  adhere  to  the  fingerprint 
material,  and  then  are  physically  lifted  out  of  the  original  crime  scene  object  by  a  sticky 
tape. 

Since  applying  chemicals  or  powders  onto  a  surface  on  which  fingerprints  reside 
changes  the  chemical  and/or  physical  composition  of  the  surface,  the  use  of  such 
“invasive”  methods  can  potentially  interfere  with  subsequent  forensic  testing  of  different 
type,  and  can  sometimes  inflict  deleterious  side  effects  on  the  surface  or  the  operator. 
Therefore,  in  the  past  30  years  several  techniques  that  can  recover  latent  fingerprints 
without  the  need  to  apply  foreign  material  directly  onto  the  fingerprints  have  been 
developed.  These  methods  use  specialized  light  sources(e.g.  Laser,  UV),  filters,  and 
detectors1'4’7'12.  They  are  very  successful  in  some  cases.  However,  like  all  other  existing 
techniques,  they  do  not  work  in  all  possible  cases  and  are  known  to  fail  completely  with 
certain  types  of  latent  fingerprints  or  object  surfaces.  As  a  result  chemical  enhancers  are 
often  reintroduced  to  aid  in  the  detection  of  latent  fingerprints1'3’9,12'19.  Further  studies 
show  that  techniques  using  special  light  sources  and  filters  work  much  better  in  more 
cases  when  combined  with  the  application  of  certain  chemicals  on  the  fingerprint  sample 
first !’9,12'19.  However  the  application  of  chemicals  directly  onto  the  fingerprints 
effectively  negates  the  non-contact  advantages  and  the  composite  methods  reverts  back 
into  invasive  methods. 

It  has  been  known  by  experienced  law-enforcement  officers  that  by  varying  the 
angle  of  a  flashlight  shining  onto  a  surface  suspected  to  bear  latent  fingerprint,  one  can 
potentially  locate  latent  fingerprints  that  are  otherwise  difficult  to  see7.  However,  in  order 


Page  54 


to  “lift”  the  latent  fingerprints  in  a  form  that  can  be  documented  and  presented  as 
evidence  in  a  court  trial,  some  “invasive”  enhancement  treatments  are  usually  considered 
necessary.  Pfister9,20  devised  an  optical  method  that  uses  a  semi-transparent  mirror  that 
can  project  light  onto  a  surface  at  a  right  angle,  while  at  the  same  time  allowing  a  camera 
or  observer  to  view  the  surface  at  a  right  angle.  A  smooth  surface  is  expected  to  appear 
bright  due  to  strong  specular  reflection,  while  a  fingerprint  mark  would  appear  darker  due 
to  much  less  specular  reflection.  Lennard  and  Margot9  21  reported  that  such  method  works 
better  when  the  sample  is  pretreated  with  cyanoacrylate.  The  use  of  only  a  right  angle  in 
Pfister’s  method  sacrificed  quite  a  bit  of  flexibility  and  polarization  based  techniques  can 
not  be  applied  to  further  improve  the  contrast  because  specular  reflection  observed  at 
right  angle  from  the  surface  is  not  preferentially  polarized  by  the  surface.  It  is  widely 
known  that  specularly  reflected  light  from  dielectrics  would  be  partially  polarized  at 
certain  range  of  viewing  angles.  However,  to  the  best  of  our  knowledge,  no  known 
application  of  polarization  has  been  reported  for  latent  fingerprints,  with  the  exception  of 
using  a  polarizing  filter  to  remove  glare  when  taking  pictures,  which  is  considered  a 
standard  photographic  technique.  Menzel  briefly  mentioned  the  possibility  of  using 
optical  polarization  to  enhance  visibility  of  latent  fingerprints  left  on  glass  but  it  appears 
that  no  further  development  took  place. 

We  propose  here  a  new  method  which  allows  the  detection  and  “lifting”  of  latent 
fingerprints  into  clearly  identifiable  digital  images  without  the  application  of  chemical 
treatments  or  indeed,  without  any  physical  contact  with  the  surface  and  fingerprint 
material.  Rather  than  employing  extraneous  material,  our  method  takes  advantage  of  the 
optical  properties,  particularly  those  related  to  specular  reflection  and  polarization,  of  the 
latent  fingerprint,  which  usually  consists  of  tiny  ridges  of  skin  residue  material  including 
sweat  (salty  water),  grease,  and  lipid2’3’23,  all  of  which  are  rather  transparent  dielectric 
materials,  making  them  difficult  to  detect  under  most  viewing  conditions.  Our  method  is 
also  applicable  to  latent  fingerprints  left  on  a  smooth  but  pliable  dielectric  surface.  The 
recovered  fingerprint  images  have  comparable  or  better  quality  to  those  obtained  by 
conventional  methods.  Fig.  l~Fig.  10  illustrate  the  fingerprint  detection  and  lifting 
capabilities  of  our  new  methods  applied  to  several  common  items  and  surfaces.  Note  that 
our  method  is  not  restricted  to  viewing  the  surface  at  a  right  angle,  and  our  use  of 


Fig.  1  Picture  of  the  three  sample  items  bearing  latent  fingerprints:  a  hard  cover  book,  a  plastic  CD  case  with 
underlying  insert  patterns,  and  a  stainless  steel  blade  of  a  Swiss  army  knife. 
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Fig.  2  Close  up  view  of  the  hard  cover  book  bearing  latent  fingerprint  under  normal  viewing 
condition.  Note  that  this  image  has  undergone  digital  linear  contrast  enhancement  but  still  the 
fingerprint  mark  is  not  visible. 


Fig.  4  The  same  area  of  the  hard  cover  book  as  in  Fig.  2  but  taken  with  our  specially  arranged 
specular  lighting  condition  plus  polarization  image  processing.  The  latent  fingerprints  are  revealed 
and  at  the  same  time  the  background  pattern  from  the  book  title  is  greatly  suppressed. 


Fig.  5  Close  up  view  of  the  plastic  CD  case  with  insert  pattern  under  normal  viewing  condition.  No 
fingerprint  is  visible  although  the  image  has  undergone  digital  linear  contrast  enhancement. 
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Fig.  6  The  same  area  of  the  plastic  CD  cover  with  insert  pattern  as  in  Fig.  5  but  taken  with  our 
specially  arranged  specular  lighting  condition  with  three  different  polarizer  orientation  and  then  the 
degree  of  polarization  image  computed.  The  latent  fingerprints  are  revealed. 


Fig.  7  The  same  area  of  the  plastic  CD  cover  with  insert  pattern  as  in  Fig.  5  but  taken  with  our 
specially  arranged  specular  lighting  condition  plus  polarization  image  processing.  The  latent 
fingerprints  are  revealed  and  at  the  same  time  the  background  pattern  from  the  CD  insert  is  greatly 
suppressed. 
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Fig.  9  The  same  area  of  the  stainless  steel  Swiss  army  knife  as  in  Fig.  8  but  taken  with  our  specially 
arranged  specular  lighting  condition  plus  polarization  processing.  The  latent  fingerprints  are 
revealed  in  the  degree  of  polarization  image. 
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Fig.  10  The  same  area  of  the  stainless  steel  Swiss  army  knife  as  in  Fig.  8  but  taken  with  our  specially 
arranged  specular  lighting  condition  plus  polarization  image  processing.  The  latent  fingerprints  are 
revealed  and  at  the  same  time  the  background  pattern  from  the  book  cover  is  greatly  suppressed. 


Non-contact  optical  latent  fingerprint  enhancement  and  lifting 
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Fig.  11  Schematic  of  physical  principles  concerning  non-contact  latent  fingerprint  enhancement  and 
lifting,  (a)  Macroscopic  reflection  from  a  surface  consists  of  two  distinct  kinds,  i.e.  specular  and 
diffuse.  For  specular  reflection  the  angles  of  incidence  and  reflection  are  equal,  while  for  the  diffuse 
case  the  reflected  intensity  may  approximately  have  an  effectively  uniform  distribution  over  all 
directions  in  a  hemisphere.  Most  surfaces  exhibit  both  types  of  reflections,  but  one  type  may  be 
stronger  than  the  other,  (b)  Partial  polarization  of  specularly  reflected  light  from  a  semi-transparent 
dielectric  surface.  It  is  known  that  the  light  reflected  from  the  smooth  surface  is  partially  polarized 
with  the  polarization  being  perpendicular  to  the  plane  of  reflection,  (c)  Live  human  skin  is  kept  soft 
and  pliable  by  the  constant  oily  secretion  of  hypodermic  glands.  The  ridge  area  of  the  skin  pattern 
tends  to  leave  a  dielectric  residue  on  a  surface  touched  by  a  finger,  (d)  The  residue  left  in  (c)  forms 
the  latent  fingerprint.  Using  a  method  that  generates  a  sufficient  contrast  difference  between  the 
latent  fingerprint  and  the  rest  of  the  surface  in  the  camera  image,  a  successful  detection  and 
extraction  can  be  achieved  without  applying  physical  or  chemical  treatments  to  the  surface.  Note  that 
the  camera  position  in  (d)  is  oriented  in  such  a  way  that  it  captures  the  specular  component  of 
reflected  light  from  the  clean  surface  only  while  the  specular  reflection  component  of  the  residue  is 
not  captured.  Additionally,  when  a  finger  touches  a  pliable  dielectric  surface  it  could  cause  a  plastic 
print  on  the  surface.  In  this  case  the  difference  in  the  surface  normal  caused  by  the  plastic  print 
ridges  will  serve  the  same  purpose  of  creating  contrast  in  intensity  and  polarization  under  proper 
lighting. 

We  start  with  the  principal  physical  and  physiological  basis  of  our  non-contact  optical 
latent  fingerprint  enhancement  and  lifting  in  this  section.  The  more  detailed  formulas 
used  in  our  computation  will  follow  later.  The  physics  underlying  the  method  is 
illustrated  in  Fig:  IT When' a  finger  touches  theobject  surface;  a  dielectric residue  "marlr^ 
bearing  the  fingerprint  pattern  is  imprinted  on  it.  The  residue  on  the  surface  induces 
differences  in  optical  polarization  or  reflection  or  both  between  the  clean  part  of  the 
surface  and  that  bearing  the  print.  The  optical  information  is  captured  and  enhanced  by 
our  unique  optical  setup  and  stored  as  digital  images.  Further  digital  processing  of  the 
captured  images  enables  us  to  “develop”  or  “lift”  the  latent  fingerprint  pattern  without 
applying  any  powder  or  other  chemicals  to  the  object.  Our  optical  setup  is  designed  on 
the  basis  of  the  well  known  Fresnel  reflection  theory  for  orthogonal  polarizations  and  the 
theories  for  macroscopic  surface  reflectance  developed  for  computer  vision  and  graphics. 
As  an  aside,  it  is  interesting  to  note  that  biologists  and  zoologists  have  found  that  certain 
animal  species  have  visual  systems  that  sense  and  utilize  (in  or  near)  visible  light’s 
polarization  in  the  natural  environment,  e.g.  backswimmer  Notonecta  glauca  can  detect 
the  polarization  of  light  reflected  from  smooth  water  surfaces  and  use  it  to  land  and 
plunge  safely  on  the  water  surface24,25.  Indeed,  our  original  step  to  design  our  optical 
setup  for  latent  fingerprint  detection  was  inspired  from  this  ability  of  Notonecta  glauca  in 
detecting  the  surface  of  the  water. 
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In  Fig.  1 1(c),  we  illustrate  a  cross  section  view  of  the  fingerprint  on  a  surface.  The 
ridge  area  corresponds  to  a  small  amount  of  residue  on  the  surface,  while  the  furrow  area 
does  not.  All  existing  enhancement  methods  take  advantage  of  this  situation  by  applying 
materials  that  selectively  attach  to  or  interact  with  only  the  residue  area  and  produce  a 
colored  or  fluorescent  pattern  of  the  residue  area.  Our  non-contact  method  exploits  this 
situation  in  a  different  way  (Fig.  11(d)),  with  a  common  household  light  source 
(incandescent  or  fluorescent,  does  not  really  matters  here),  a  camera  and  the  surface 
being  inspected  arranged  in  such  a  way  that  the  geometry  conforms  approximately  to  the 
law  of  (specular)  reflection.  Thus,  the  incident  angle  of  light  from  the  source 
approximately  equals  the  viewing  (reflection)  angle  of  the  camera,  so  that  the  camera  will 
capture  the  light  reflected  specularly  from  the  non-residue  area,  and  also  only  the  light 
reflected  diffusely  from  the  residue  laden  area.  The  reason  is  that  the  residue  stain  area  is 
likely  to  have  different  surface  normal  directions  and  indices  of  refraction  as  compared 
with  the  uniform  or  smoothly  varying  surface  normal  direction  of  the  unstained  surface 
area.  The  highly  localized  nature  of  specular  reflection  energy  distribution  makes  it  very 
sensitive  to  even  minute  changes  in  the  direction  of  the  surface  normal.  Since  the 
specular  reflection  component  is,  in  general,  much  stronger  than  the  diffuse  reflection 
component26  (see  Fig.  11(a)),  one  potentially  finds  an  enhanced  contrast  between  the 
residue  laden  ridge  mark  and  the  clean  surface  furrow  ‘negative-mark’.  Another  often 
encountered  case  is  a  plastic  fingerprint  left  on  a  pliable  dielectric  surface.  There  may  or 
may  not  be  biological  residue  left  on  the  surface  but  the  ridge  and  furrow  patterns  formed 
by  the  pliable  dielectric  material  itself  create  differences  in  surface  normal  compared  to 
the  undisturbed  surface  area  and  will  serve  the  same  purpose. 

In  some  cases  this  contrast  enhancement  from  the  specular  reflection  effect  alone 
is  insufficient.  The  object  itself  may  have  a  complicated  high  contrast  pattern  under  the 
top  coating  of  the  surface  (see  Fig.  11  (c)  and  (d))  that  interferes  with  fingerprint  pattern 
even  after  the  enhancement.  This  problem  has  been  recognized  by  many  practitioners27,28. 
Discrete  Fourier  transform  analysis  has  been  shown  to  be  able  to  remove  regular  patterns 
that  vary  periodically,  but  cannot  deal  with  a  general  background  that  is  not  periodic.  An 
additional  characteristic  of  the  specular  reflection  is  that  it  tends  to  be  partially  polarized 
in  a  plane  perpendicular  to  the  plane  of  reflection,  see  Fig.  11(b).  One  or  more 
polarization  analyzers  collecting  polarization  components  at  different  angles  can  provide 
complete  information  about  the  polarization  state  of  the  reflected  light.  Based  on  the 
polarization  information  we  can  further  extract  only  the  specular  component  of  this 
reflection  and  get  a  much  cleaner  fingerprint  image  because  for  the  most  part  the  light 
coming  from  the  pattern  beneath  the  top  coating  of  the  object  surface  is  due  mostly  to 
unpolarized,  diffuse  reflection.  We  have  also  found  that  in  some  cases  some  of  the 
polarization  images  simply  show  higher  contrast  between  the  fingerprint  and  its 
background  than  the  fingerprint  images  recovered  using  specular  reflection  alone. 

The  general  expression  for  the  observed  intensity  of  partially  polarized  light  I  as 
a  function  of  the  angle  <p  of  orientation  of  a  polarization  analyzer  can  be  written  as 
follows: 


I((p)  =  Iu+IAcos\2(e-(p)\  =  Iu{\  + pcos\_2(9-(p)§  (6) 

where  9  is  the  orientation  angle  of  the  major  axis  of  the  polarization  ellipse,  Iv  is  a  half 
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of  the  total  pixel  intensity,  and  p  =  lAUu  is  the  degree  of  linear  polarization  at  a  given 
pixel  in  a  digitized  image.  The  reference  axis  for  (p  and  6  can  be  arbitrarily  chosen. 

Since  the  exact  index  of  refraction  of  the  surface  in  our  study  is  considered  unknown, 
putting  one  polarizer  at  a  given  orientation  angle  in  front  of  the  camera  and  taking  a 
picture  cannot  provide  complete  information  about  the  polarization  state  of  the  received 
light.  By  taking  three  pictures  with  the  polarizer  oriented  at  three  different  angles,  for 
example  (p=  0,  45  and  90  degrees,  we  can  recover  Iu,  I A ,  and  6  for  each  pixel  of  the 
image  using  the  following  expressions: 

A,=U.+W2 

/,  =  ,)! +(/„-/„)’  (?) 

9  =  arctan  [( /„, -/„)/(/,„-/„  )]/2 

Here  indices  0,  45,  and  90  indicate  the  orientation  of  the  polarizer  in  degrees  when  the 
image  was  taken.  Because  9  and  9+71  are  indistinguishable  for  phase-blind  visual 
sensors  in  most  conventional  cameras,  the  meaningful  range  of  9  is  restricted  to  (0~n). 

We  usually  use  9  in  the  range  from  0  to  n.  Polarization  camera  systems  able  to  rapidly 
take  the  required  pictures  have  been  developed  by  Wolff  and  his  colleagues  '  .  The 
formulation  and  symbols  used  here  follows  from  our  previous  work32,33  and  are  slightly 
different  from  those  used  by  Wolff.  Since  the  background  object  pattern  is  most  likely 
caused  by  pigments  beneath  the  transparent  substrate  that  is  used  to  hold  them,  the  object 
pattern  intensity  signals  are  mostly  due  to  diffuse  reflection,  which  is  nearly  unpolarized 
and  thus  IA  and  sometimes  p  are  close  to  zero.  Thus,  with  our  polarization  technique  we 
can  extract  the  purely  specular  reflection  component  from  the  top  surface  by  computing 
images  of  IA  or  p  for  every  image  point.  Such  images  often  carry  a  substantial  contrast 

between  the  fingerprint  residue  pattern  and  the  clean  area  in  between.  Note  that  if  only  0 
and  90  orientation  images  are  taken,  the  fingerprint  may  still  be  enhanced  in  the 
polarization-difference  image34’38,  but  the  .Q.or  90  direction  must  be  nearly  parallel  to 
either  the  object  surface  or  the  fingerprint  ridge  surface,  which  can  be  challenging  to 
arrange  when  the  surface  is  not  flat. 

Experiments 

A  step-by-step  application  of  the  new  optical  method  is  presented  in  Fig.  1 2~Fig.  19.  Fig. 

12  gives  an  overview  of  the  experimental  setup  when  the  specially  arranged  light  is  on 
and  the  ordinary  room  light  is  turned  off.  The  surface  being  inspected  is  the  metal  casing 
of  an  electric  air  pump  (with  brand  name  sticker  “Linicon”)  which  is  painted  orange.  Fig. 

13  shows  how  the  surface  looks  like  with  the  specially  arranged  lighting  turned  off,  and 
the  ordinary  diffuse  fluorescent  room  light  on.  The  image  in  Fig.  13  is  shown  with  a 
common  digital  enhancement  available  in  most  image-processing  software,  setting  the 
brightest  pixels  in  the  image  to  the  maximum  possible  value  allowed  by  the  display,  and 
the  darkest  pixels  to  the  lowest  display  value  and  linearly  rescaling  the  rest  pixel  intensity 
values.  Fig.  14  presents  the  same  image  data  as  in  Fig.  13,  but  enhanced  instead  by 
histogram  equalization.  This  enhancement  method  remaps  the  pixel  values  according  to 
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the  histogram  distribution  of  their  magnitudes,  distributing  them  more  evenly  over  the 
dynamic  range  of  the  display  (see39).  These  two  images  illustrate  the  ‘latent’  nature  of  the 
fingerprint:  the  natural  contrast  is  so  low  that  not  only  the  unaided  human  eye  cannot 
detect  it,  but  even  widely  used  digital  image  enhancements  do  not  reveal  its  presence. 
Fig.  15  and  Fig.  16  present  images  taken  with  only  the  specular  illumination  turned  on, 
and  digitally  enhanced  with  the  same  methods  as  used  to  produce  Fig.  13  and  Fig.  14, 
respectively.  Imaging  the  light  specularly  reflected  from  the  surface  yields  a  major 
enhancement  not  achievable  with  the  digital  enhancements  alone,  an  enhancement 
traditionally  achieved  with  powders  and  chemicals,  but  completely  without  destructive 
side  effects.  Fig.  17  and  Fig.  18  are  zoomed  in  view  showing  the  detailed  quality  of  the 
recovered  fingerprint  images.  Fig.  19  is  Fig.  17  digitally  reprojected  and  rotated  to  a 
canonical  “up  right”  view,  as  if  the  picture  was  taken  with  a  camera  looking  at  right  angle 
from  the  surface  with  the  fingertip  pointing  up  for  easier  comparison  with  standard 
database  and  for  display  in  a  court. _ _ _ 


Fig.  12  Setup  Overview  for  Fig.  13~Fig.  19. 


Fig.  13  Ordinary  lighting,  linearly  scaled. 
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Fig.  15  Specular  lighting,  linearly  rescaled. 


Fig.  16  Specular  lighting,  histogram  equalized. 
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Ffg,  79  Reprojected  and  rotated  upright  yic>v  pf-F5g.  i7.  >,.r  , 

The  results  of  another  experiment  employing  our  non-invasive  optical  method  are 
illustrated  in  Fig.  20~Fig.  22.  In  this  case,  the  surface  with  the  latent  fingerprint  was  the 
paper  cover  of  an  ordinary  desk  calendar  (Fig.  20),  and  presented  a  greater  challenge  than 
the  solid  colored  metallic  surface.  The  paper  surface  contained  a  printed  pattern  whose 
light  absorption  interferes  with  the  optical  detection  of  the  fingerprint.  We  used  both 
specular  reflection  and  polarization  analysis  to  extract  the  latent  fingerprint  (Fig.  21).  For 
the  polarization  analysis,  images  were  taken  with  a  linear  polarization  analyzer  mounted 
in  front  of  the  camera,  and  oriented  at  three  different  angles.  The  picture  displayed  in  Fig. 
21  is  the  value  image  of  the  derived  quantity  IA  (see  Eq.  (7)  in  supporting  material).  We 

emphasize  here  that  Fig.  21  is  not  an  image  of  the  ordinary  intensity  distribution,  but 
rather  a  mapping  of  a  certain  physical  quantity  derived  from  the  polarization  distribution 
of  the  light  comprising  the  image.  The  specular  component  of  the  surface  reflection  is 
now  evident,  and  the  background  pattern  is  gone.  Fig.  22  shows  a  cropped,  close-up  of 
the  fingerprint  area  of  the  image  seen  in  Fig.  21.  These  results  lead  to  two  additional 
conclusions:  first,  the  non-invasive  optical  method  can  extract  latent  prints  from  some 
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paper  surfaces  as  well  as  from  smoother  surfaces;  and  second,  the  processing  of  the 
polarization  information  in  the  image  can  further  enhance  the  quality  of  the  recovered 
latent  fingerprint  under  certain  conditions. 

Sometimes  the  specular  reflection  component  can  be  obscured  by  more  intense 
diffuse  reflection.  We  applied  our  methods  to  such  a  case,  deliberately  picking  one  of  the 
strongest  diffuse  reflectors,  a  white  cotton  lining  underneath  a  soft  clear  plastic  CD  sleeve 
(Fig.  23~Fig.  26).  Fig.  23  shows  an  image  taken  without  a  polarizing  filter  in  front  of  the 
camera.  The  diffuse  white  light  is  so  strong  that  the  specular  reflection  output  can  barely 
enhance  the  latent  fingerprint  on  the  plastic  surface.  However,  the  polarization-based 
analysis  followed  by  the  histogram  equalization  readily  enhances  the  latent  fingerprint 


Fig.  20  Paper  calendar  cover  with  underlying  picture.  A  fingerprint  is  revealed  by  specular  lighting. 


Page  68 


Fig.  23  A  soft  plastic  CD  sleeve  with  white  cotton  lining  underneath,  seen  with  ordinary  lighting. 


Fig.  24  Polarization  IA  image.  The  latent  fingerprint  on  the  CD  sleeve  is  exposed  in  high  contrast. 
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Fig.  26  Reprojected  and  rotated  Fig.  25  to  up  right  view. 

The  sticky  side  of  a  tape  has  traditionally  caused  trouble  with  invasive  methods, 
especially  methods  that  apply  powder  to  the  surface.  Most  powders  and  reagents  can  stick 
easily  anywhere  on  the  sticky  surface,  not  just  the  fingerprint  area.  In  contrast,  sticky  side 
of  the  tape  is  ideally  suited  to  our  method.  The  “sticky”  material  is  a  thin  coat  of  pliable 
semi-transparent  dielectric  that  fits  our  surface  model  perfectly.  Whether  the  latent 
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fingerprint  mark  is  formed  by  skin  residue  or  by  a  plastic  mark  formed  on  the  “sticky” 
surface,  our  surface  model  predicts  that  high  contrast  intensity  or  polarization  images  can 
be  formed  with  proper  lighting.  We  tested  our  method  on  a  piece  of  transparent  packing 
tape  and  the  results  are  very  good  (Fig.  27).  An  example  of  plastic  fingerprint  mark 
detection  using  our  method  is  shown  in  Fig.  28. 


Fig.  27  Left:  fingerprint  found  on  the  sticky  side  of  the  tape  using  specular  reflection.  Right: 
fingerprint  found  on  the  sticky  side  of  the  tape  using  polarization.  The  image  is  the  IA  image.  Note 
that  no  ordinary  digital  contrast  enhancement  is  used  on  both  images. 
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Fig.  28  Plastic  fingerprint  mark  revealed  on  a  piece  of  hardened  epoxy  resin  in  the  degree  of 
polarization  (p)  image.  No  ordinary  digital  contrast  enhancement  is  used  on  this  image. 


Theory  of  the  application  of  specular  reflection  to  latent 
fingerprints 

Modeling  surface  reflection  on  a  microscopic  scale  is  complicated  and  depends  heavily 
on  the  detailed  knowledge  of  the  molecular  material  composition  of  the  surface  material. 
However,  macroscopically,  a  more  general  model  can  be  used  that  applies  to  a  wider 
range  of  surfaces  without  the  need  for  details  about  the  surface  with  acceptable  reduction 
in  accuracy.  This  is  desirable  in  many  practical  applications,  notably  in  computer  vision 
and  graphics,  where  the  details  of  the  chemical  and  physical  composition  of  surfaces  are 
not  known  or  are  not  ^of  vital  importance.  ^Beginning  with  Refs40!41,  many  surface 
reflection  models  based  not  on  the  exact  chemical  composition  but  rather  on  a  plausible 
statistical  model  of  the  surfaces  were  proposed,  see  e.g.  Refs42,43.  A  review  of  various 
models  can  be  found  in  Refs44,45.  Because  in  our  applications  we  intend  to  extract  the 
fingerprint  without  using  any  chemical  analysis,  the  possibility  of  knowing  the  properties 
of  surface  material  beforehand  is  excluded.  However,  a  simple  model  that  describes  a 
general  trend  is  good  enough,  because  the  ultimate  form  required  for  a  fingerprint  image 
is  that  of  binarized  black  and  white  regions  separating  the  ridge  and  furrow  areas.  There 
is  no  need  to  recover  or  to  predict  the  exact  brightness  differences  in  the  gray-level 
images  taken  for  the  purpose  of  recovering  fingerprint  marks.  The  simple  Phong  model43 
and  Lambertian  model40,  both  widely  used  in  many  computer  vision  and  graphics 
algorithms,  satisfy  these  purposes. 

Macroscopically,  two  well  known  general  types  of  reflection  can  be  named.  The 
Lambertian  model  describes  a  surface  producing  perfectly  diffuse  reflection  as 

I  —  Ipkd  cos#  =  I  pkd  («•/)  (8) 
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where  /  is  the  intensity  of  the  image  point  sensed  by  the  camera,  /  is  the  point  light 
source’s  intensity,  kd  is  the  reflection  coefficient  (either  for  a  particular  wavelength  or 
for  a  particular  camera’s  spectral  response),  0  is  the  angle  between  the  surface  normal  h 
and  the  unit  vector  /  in  the  direction  of  the  light  source  as  viewed  from  the  point  of 
reflection  (see44).  Note  that  the  diffusely  reflected  light  has  the  same  intensity  for  all 
viewing  directions. 

Another  type  of  reflection  is  that  of  highlights,  or  mirror  like  reflection  observed  on  many 
smooth  surfaces.  A  more  subdued  version  is  usually  called  “sheen”.  The  Phong  model  is 
given  by 


h  =  LA°dx  +  fJPx lkj°dx  cosd  +  W (0) cos"  a] 

where  \  is  the  wavelength  of  the  light,  subscript  ‘.a’  denotes  ambient  light  source, 
subscript  ‘p’  denotes  point  light  source,  subscript  ‘d’  denotes  a  diffuse  reflection 
component,  the  new  symbol  O  denotes  color  components  in  human  and  digital  color 
vision  components,  faU  is  the  inverse  square  of  decay  distance  of  a  point  light  source 

intensity,  W(0 )  is  the  diffuse  reflection  coefficient  of  the  surface  with  a  point  light 

source  angle  of  incidence  0,  a  is  the  angle  between  the  exact  view  direction  predicted 
by  the  law  of  reflection  and  the  actual  view  direction,  as  shown  in  Fig.  11(d).  For  the 
current  study,  the  most  important  information  derived  from  (9)  is  that  the  intensity 
depends  on  a  as  a  cos"  a .  This  gives  a  simple  way  to  model  a  rapid  decay  in  intensity  if 
the  view  angle  is  different  from  that  predicted  by  the  law  of  reflection.  This  term  suggests 
that  if  the  camera,  the  light  source,  and  the  surface  being  inspected  are  arranged  in  a  way 
predicted  by  the  reflection  law  (see  Fig.  11(d)),  the  image  point  for  the  original  smooth 
surface  without  skin  residue  will  show  intensity  typical  for  specular  reflection,  while  the 
area  with  skin  residue  will  have  much  less  specular  reflection  due  to  the  slight  change  in 
the  direction  of  surface  normal  caused  by  the  skin  residue.  The  more  mirror-like  a  surface 
can  be  modeled,  the  larger  power  of  cosine  decay  it  exhibits,  which  means  better  contrast 
in  our  specular-reflection  —based  latent  fingerprint  detection  and  lifting  technique. 
Providing  all  other  factors  being  equal,  the  intensity  of  the  specular  reflection  component 
is  in  general  much  stronger  than  the  diffuse  reflection  component.  Although,  this 
statement  is  not  always  true,  it  has  been  widely  accepted  as  a  good  rule  of  thumb  in  the 
majority  of  practical  situations26.  Since  specular  reflection  has  a  tendency  of 
concentrating  reflected  energy  in  a  small  solid  angle,  as  opposed  to  the  diffuse  reflection 
which  spreads  all  the  reflected  energy  into  a  full  hemisphere,  the  same  amount  of 
reflected  energy  will  result  in  a  much  greater  flux  density  in  specular  reflection  and  thus 
in  image  brightness.  The  specular  reflection  tends  to  be  reflected  only  once  from  the 
smooth  surface,  while  the  diffuse  reflection  gives  light  that  experienced  a  multiple 
scattering  inside  the  surface  before  re-emerging.  Each  scattering  only  weakens  the 
intensity  but  seldom  enhances  it.  Wolff26  experimentally  measured  the  ratio  between 
specular  and  diffuse  reflection  intensities  for  several  different  surfaces  and  reported  ratios 
varying  from  about  150:1  to  250:1.  For  many  digital  sensors  with  8-bit  brightness 
resolution,  this  is  close  to  the  maximum  intensity  ratio  of  255:1.  This  gives  a  strong 
support  to  our  main  assumption,  that  the  specular  reflection  component  is  in  general 
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stronger  than  the  diffuse  one.  Our  experiment  results  so  far  also  support  the  validity  of 
such  assumption. 

In  Ref26  another  important  theoretical  result  relating  to  fingerprint  detection  issue  was 
reported,  i.e.:  if  the  reflection  coefficient  of  the  diffuse  component  is  about  1/33  more 
than  the  reflection  coefficient  of  the  specular  component,  the  diffuse  component  can  be 
stronger  or  at  least  comparable  to  the  specular  component.  If  the  underlying  surface 
consists  of  complicated  patterns  similar  in  strength  and  spatial  frequency  to  the  latent 
fingerprint  pattern  on  top  of  it,  the  method  based  on  a  purely  specular  reflection  is  not 
satisfactory.  This  is  the  point  where  the  polarizer  should  be  used. 

The  behavior  of  specular  component  is  governed  by  the  well-known  Fresnel  reflection 
coefficients  formula46: 


V  E0i  Jy 


Or 

.  E0i  J\\ 


_  nt  cos 6j  - nt  cos 9t  _  sin(6(  -6t) 
ni  cos  Qt  +  n,  cos  9t  sin(6/  +  6t ) 

_  nt  cos  6i  -  ni  cos  9t  _  tan(#.  -  9t) 
n:  cos 9t  +  nt  cos 9i  tan {9t  +9t) 


(10) 


where  subscripts  ‘i’,  ‘t’,  and  ‘r’  stand  for  incident,  transmitted,  and  reflected  component. 
The  subscripts  ‘  ±  ’  and  ‘  ||  ’  are  related  to  the  plane  of  incidence.  In  Fig.  1 1(b)  the  plane  of 
this  paper  is  the  plane  of  incidence.  In  case  of  the  specular  reflection  it  contains  both  the 
incident  and  reflected  light  wave  vectors.  It  is  well  known  (see  e.g.,46)  that  ^  can  be 

exactly  zero  at  the  Brewster’s  angle  9B ,  which  is  given  as 


(11) 

", 

If  light  is  incident  from  the  air,  >  ,  while  '  varies  from  1.4  to  2.0  for  most  of 

dielectrics  in  the  visible  band  (wavelength  of  about  4pQ~700.=nm)4  ,..Eq.  (1 1)  shows  that 
the  corresponding  Brewster’s  angles  vary  from  55  to  74  degrees,  respectively.  Although, 
there  are  certain  materials  with  higher  refractive  index46,  we  confine  our  discussion  to  the 

above-mentioned  range  of  n‘ .  Therefore,  when  we  consider  the  angle  of  incidence 
between  55  and  74  degrees,  the  reflected  light  is  highly  partially  polarized  with  the  plane 
of  polarization  perpendicular  to  the  plane  of  incidence.  This  case  is  referred  to  as 
“horizontal  polarization”  with  respect  to  the  surface  being  inspected.  So  far  we  have  only 
discussed  dielectric  surfaces  which  are  adequate  in  most  cases  since  “pure”  metallic 
surfaces  are  rather  rare  in  everyday  life.  Pure  metal  surfaces  are  oxidized  quickly  and  the 
actual  layer  “responsible”  for  specular  reflection  is  often  either  the  oxides  on  the  surface 
or  the  protective  painting  layer  which  is  also  a  dielectric  material.  In  fact,  a  lot  of 
metallic-looking  merchandise  today  are  actually  coated  with  highly  reflective  dielectric 
paints.  In  cases  when  the  underlying  pure  metal  reflects  more  light  than  the  upper 
dielectric  coating,  the  proposed  method  may  not  work. 

Discussion 
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The  currently  popular  latent  fingerprint  detection  and  extraction  methods  used  by  law 
enforcement  agencies  include,  but  are  not  limited  to,  powdering,  Sudan  black  staining, 
iodine  fuming,  ninhydrin  (sometimes  followed  by  further  enhancement  with  zinc 
chloride)  and  DFO  application,  silver  nitrate  development,  cyanoacrylate  (glue)  fuming, 
gentian  violet  staining,  vacuum  metal  deposition,  laser  excited  luminescence,  and  RUVIS 
(Reflected  Ultra  Violet  Imaging  System)1"4.  While  this  list  may  seem  long,  there  is  still 
need  for  new  methods,  because  every  existing  method  tends  to  be  unsuitable  for  some 
surfaces,  due  either  to  its  inadequacy  in  lifting  the  print  from,  or  to  its  damaging  side 
effects  to  the  surfaces.  In  particular,  the  chemical  and  physical  processing  directly  applied 
to  the  fingerprint  bearing  surface  in  order  to  extract  latent  fingerprints  can  potentially 
inflict  deleterious  effects  upon  the  fingerprint,  the  operator,  and/or  the  object  surface 

1  1  If)  IQ  *  .  .  . 

being  examined  '  ’  ’  .  For  example,  the  iodine  vapor  in  the  iodine  fuming  method  is 
highly  corrosive  and  toxic2.  Thus,  in  practice,  often  valuable  and/or  irreplaceable  objects 
are  not  searched  for  fingerprint  at  all2’3’7,  except  in  a  few  major  cases  involving  extremely 
serious  crimes.  Furthermore,  the  chemicals  used  to  enhance  fingerprint  contrast  or  to 
induce  luminescence  may  need  long  processing  time,  are  sometimes  toxic, 
environmentally  unfriendly,  or  radioactive.  The  process  to  speed  up  the  chemical  reaction 
can  be  dangerous,  e.g.  sodium  hydroxide  used  to  speed  up  cyanoacrylate  fuming  can 
generate  extreme  heat  if  the  two  come  into  contact4.  Samples  are  often  baked  to  high 
temperature  after  many  fuming  procedure  to  speed  up  the  print  development.  Chemicals 
used  for  fingerprint  enhancement  can  often  be  harmful  to  the  operator  if  not  handled 
correctly  using  proper  procedure  and  protective  equipments  because  they  are  designed  to 
react  with  or  adhere  to  the  fingerprint  residues,  which  are  the  same  material  found  on 
human  skin3.  Some  chemicals  require  specific  solvents  that  have  undesirable  side  effects; 
e.g.,  the  phenol  in  the  solution  for  Gentian  violet  is  highly  caustic  and  poisonous2. 

Lasers  induced  fluorescence  is  one  of  the  first  optical  methods  for  lifting  latent  prints, 
utilizing  induced  luminescence  of  the  fingerprint  material1"3,10.  However,  there  are  several 
factors  limiting  its  use.  The  natural  fluorescence  signal  (without  the  prior  application  of 
strongly  fluorescent  chemical  or  powder)  is  in  general  very  weak.  Thus  the  laser  used 
must  operate  in  a  specific  frequency,  and  must  have  a  high  enough  power  rating.  Such  a 
laser  device  is  fairly  expensive  and  bulky  (due  to  power  and  cooling  requirements),  and 
thus  it  can  be  deployed  only  to  a  few  large  and  well  funded  crime  labs  and  not  easily 
made  portable  outside  a  dedicated  crime  lab18.  Less  powerful  laser  or  arc  lamp  with  filter 
can  be  used  as  substitute  but  will  produce  much  reduced  effectiveness.  As  a  result  these 
alternative  systems  need  chemical  enhancements.  The  fact  that  many  commonly 
encountered  fingerprint  laden  objects  found  at  a  crime  scene  contain  organic  substances 
that  also  fluoresce  when  excited  by  a  laser  often  causes  significant  background 
noise  Thus,  laser-excited  luminescence,  like  other  existing  methods,  cannot  be 

applied  to  certain  types  of  surfaces.  It  has  also  been  found  that  the  signal  strength  of  the 
natural  fluorescence  of  fingerprints  varies  greatly  from  person  to  person  and  even  from 
time  to  time  of  the  same  person10.  As  a  result,  in  real  applications  laser-excited 
luminescence  techniques  are  more  often  used  with  the  aid  of  applying  fluorescence 
enhancing  chemicals  to  get  better  and  more  consistent  results  ’  ’  ’  "  .  However  the  use 
of  chemical  fluorescence  enhancers  negates  the  non-contact  advantage  and  the  combined 
method  reverts  back  to  an  invasive  method. 

RUVIS  is  pioneered  by  the  research  at  the  National  Police  Agency  of  Japan  and  was 
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originally  targeted  for  enhancing  cyanoacrylate-developed  latent  prints.  It  is  found  that 
the  cyanoacrylate-developed  latent  prints  that  are  translucent  under  visible  light  become 
opaque  under  UV.  Latent  prints  deposited  in  sebaceous  matter  or  oily  residue  can 
sometimes  be  detected  by  RUVIS  before  the  application  of  enhancing  materials.  The  oil 
strongly  absorbs  UV  and  show  up  as  dark  patterns  under  UV  light  and  detector.  RUVIS 
can  sometimes  detect  fingerprints  up  to  a  year  old  in  purely  non-contact  style1.  The 
drawback  is  the  need  of  expensive  specialized  UV  light  source  and  sensors.  UV  light  can 
be  harmful  to  both  human  eyes  and  skin  so  proper  precaution  and  protective  gear  is 
necessary.  Bramble  et  al.47  proposed  using  laser  in  UV  band  to  induce  luminescence  that 
is  also  in  UV  band.  Although  UV  laser  is  not  considered  invasive,  they  find  that  the 
luminescence  property  of  the  fingerprint  material  decrease  significantly  after  being 
exposed  to  UV  laser  for  extended  period  of  time47. 

The  “Episcopic  Coaxil  Illumination”  method  proposed  by  Pfister20  use  the  intensity 
difference  between  specular  and  diffuse  reflection  to  enhance  the  visibility  of  latent 
fingerprint.  The  design  always  looks  at  the  surface  from  the  right  angle.  The  advantage  is 
that  the  picture  will  always  appear  in  frontal  view  suitable  for  archiving.  However  with 
the  advent  of  digital  image  processing,  oblique  view  can  be  easily  reprojected  back  to 
frontal  view  (see  Fig.  19  and  Fig.  26)  so  it  is  not  that  crucial.  We  have  shown  that  by 
varying  observation  angles  we  can  get  better  contrast  in  some  cases.  We  have  also  shown 
that  the  partially  polarized  nature  of  oblique  specular  reflection  can  significantly  enhance 
the  contrast  of  latent  fingerprint  and  at  the  same  time  suppress  the  interfering  background 
pattern.  Viewing  the  surfaces  only  at  the  right  angle  preclude  the  use  of  polarization 
because  specular  reflection  at  the  right  angle  is  not  partially  polarized  by  the  surface. 
There  is  a  commercial  product  called  “fingerprint  camera”  that  has  existed  for  a  long 
time4.  Some  may  confuse  it  as  yet  another  non-contact  optical  method  for  the  detection  of 
latent  fingerprints.  However,  this  device  is  in  fact  an  ordinary  camera  customized  to 
record  non-exemplar  fingerprints  that  have  been  enhanced  by  other  methods  or  for 
“patent  fingerprints”  that  is  already  visible.  The  customization  include  dedicated  lighting 
to  reduce  uneven  lighting,  shadow,  and  glare;  a  fixed  object  distance  and  fixed  focus, 
aperture,  exposure  time,  ...etc.  These  preset  camera  settings  enable  law  enforcement 
officers  who  may  not  be. photographic  experts  to  be. able  to. take  good  fingerprint  pictures 
consistently  for  court  use.  Such  a  “fingerprint  camera”  does  not  have  any  “contrast 
enhancement”  or  “detection”  capability  for  hidden  latent  fingerprints. 

The  novel  optical  method  we  propose  here  is  capable  of  recovering  high  quality  digital 
images  of  hidden  latent  fingerprints  without  the  application  of  any  chemicals  or  physical 
contacts  with  the  examined  object.  Like  any  other  existing  latent  fingerprint  enhancement 
technologies,  our  method  has  its  limitations.  Our  method  is  designed  to  take  advantage  of 
the  intensity  and  polarization  differences  between  specular  and  diffuse  reflections  so  it 
will  work  best  on  a  relatively  smooth  and  non-porous  surface.  The  chance  of  detecting 
fingerprints  on  a  highly  porous  and  absorbing  surface  like  certain  kinds  of  paper  using 
our  method  is  fairly  low.  However  our  method  possesses  unique  advantages  over  existing 
methods  on  smooth  surfaces  in  that  complex  patterns  on  the  surface  can  be  easily 
suppressed  optically  using  our  polarization  method.  Note  that  while  the  most  common 
use  of  photographic  polarization  filter  is  to  remove  glare,  our  use  of  the  polarizer  is  quite 
the  opposite.  Our  polarization  method,  in  a  sense,  extracts  only  the  specular  “glare”  that 
was  usually  treated  as  a  mere  nuisance.  Specifically  we  discovered  that  inside  the 
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specular  “glare”  under  certain  conditions  a  high  contrast  clean  latent  fingerprint  image  is 
present. 

Menzel22  briefly  mentioned  the  possibility  of  using  polarization  to  detect  fingerprint  on 
glass.  We  have  shown  that  polarization  can  be  used  to  enhance  fingerprint  visibility  on  a 
wide  variety  of  surfaces,  not  just  on  glass.  We  also  show  that  polarization  can  be  used  to 
remove  interfering  background  pattern  that  has  not  been  mentioned  before.  Another 
advantage  of  our  method  is  that  our  method  is  less  affected  by  the  degradation  of  latent 
fingerprint  over  time  compared  to  methods  that  relies  on  chemical  reactions  with  the 
organic  residues  or  water  in  the  latent  fingerprint  marks.  Over  time  an  exposed  latent 
fingerprint  mark  will  lose  its  water  contents  via  evaporation,  and  the  amino  acid 
components  will  degrade  and  its  chemical  properties  change1-4.  Since  our  method  does 
not  rely  on  water  or  detailed  chemical  composition  of  the  fingerprint  mark,  our  method  is 
less  likely  to  be  affected  by  chemical  decomposition.  As  long  as  the  geometric  difference 
in  the  surface  remains,  our  method  can  be  used  effectively.  The  example  pictures  shown 
were  taken  up  to  several  weeks  after  the  fingerprint  is  made  (and  kept  in  our  lab/office 
environment  undisturbed)  and  they  show  little  degradation  of  quality. 

Because  of  a  great  variety  of  the  latent  fingerprints  that  can  potentially  be  found  in  very 
different  crime  scene  all  over  the  world,  no  single  latent  fingerprint  enhancement  method 
can  handle  all  possible  cases.  Since  every  method  targets  different  physical  and  chemical 
properties  of  different  kinds  of  fingerprints,  it  is  often  found  that  applying  more  than  one 
method  on  the  same  surface  will  unveil  different  fingerprints  for  different  method,  e.g.  18. 
Thus  the  introduction  of  a  new  latent  fingerprint  detection  method  that  uses  different 
physical  principles  than  those  used  by  existing  methods  and  one  that  will  not  interfere 
with  other  methods  should  always  be  beneficial  to  law  enforcement  efforts. 

Conclusion 

We  have  introduced  a  novel  optical  method  to  detect,  enhance  and  lift  the  latent 
fingerprints  that  are  otherwise  difficult  to  see.  Such  method  is  non-invasive  and  so  will 
not  interfere  with  other  forensic  examinations  and  will  not  inflict  deleterious  side  effects 
on  the  surface.  The  equipment  required  is  much  cheaper  than  most  other  non-contact 
methods  proposed  so  far.  The  new  method  is  not  applicable  to  highly  porous  and 
absorbing  surfaces  but  works  well  on  most  other  surfaces.  The  new  method  also  works  on 
the  sticky  side  of  the  tape,  which  is  a  particularly  tricky  surface  for  many  invasive 
methods.  The  recovered  latent  fingerprint  image  has  very  good  quality  compared  to  other 
existing  method.  Since  it  does  not  interfere  with  other  methods  it  is  advisable  to  try  this 
method  before  other  methods. 
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Abstract:  Polarization  of  light  carries  much  useful  information  about  the 
environment.  Several  biological  studies  show  evidences  of  certain  animal  species 
using  polarization  information  for  navigation  and  other  purposes.  It  has  been 
shown  that  bio-inspired  Polarization  Difference  Imaging  technique  can  be  used  to 
facilitate  target  detection  and  feature  extraction  in  scattering  media.  It  has  been 
proven  that  “Polarization  Sum”  and  “Polarization  Difference”  signals  are  the 
optimum  linear  combination  for  a  scene  with  a  uniform  probability  distribution  of 
polarization  directions.  However  in  many  real  environments  the  scene  has  non- 
uniform  probability  distribution  of  polarization  directions.  Using  principal 
component  analysis  on  statistical  scene  data  we  developed  a  method  to  adaptively 
determine  the  two  optimum  information  channels  that  can  be  formed  as  a  linear 
combination  of  the  signals  of  two  original  polarization  channels.  We  also 
determined  the  optimal  orientations  of  polarization  filters  that  best  achieve 
separation  of  a  target  from  the  background,  where  the  target  is  defined  as  an  area 
with  distinct  polarization  characteristics  as  compared  to  the  background. 
Experimental  results  and  simulations  confirm  that  adaptive  polarization  difference 
imaging  outperforms  the  polarization  difference  imaging  with  fixed  channels  in 
most  situations. 
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Polarization  is  an  intrinsic  feature  of  light  that  provides  valuable  information  about  a 
scene  beyond  that  provided  by  the  scene’s  spectral  (color)  and  intensity  distributions. 
Polarized  light  has  been  studied  extensively  since  Fresnel’s  works  on  the  wave  theory 
and  the  polarization  important  issue  in  many  areas  of  modem  technology.  There  are 
many  books  about  polarized  light  (see,  e.g.,  [48,  49]).  Polarization  information  has 
proven  useful  in  several  fields,  including  computer  vision,  robotics,  target  recognition, 
and  material  classification  (see,  e.g.,  [50,  51,  52,  53,  54,  and  55]).  Polarization 
parameters,  e.g.,  Stokes  parameters,  are  in  general  more  sensitive  to  the  nature  of  a 
scattering  surface  than  the  total  intensity  is  [56].  In  such  case,  polarization  imaging 
techniques  offer  the  possibility  of  producing  images  having  higher  inherent  visual 
contrast  than  conventional  image  processing  of  the  intensity  distribution  [50],  Much  of 
work  has  been  devoted  to  target  (particularly,  small  target)  detection  utilizing 
polarimetric  properties  of  media  has  been  done  (see,  e.g.,  [56,  57,  58]). 

The  polarization  of  light  is  not  discernible  to  the  unaided  human  eyes  [59],  but  the 
polarization  has  been' shown  to  provide  valuable  information  to  other  species.  In  1949, 
Nobel  laureate  Karl  von  Frisch  established  that  honeybees  through  their  perception  of 
polarized  light  use  the  sun  as  a  compass  through  their  perception  of  the  polarization 
pattern  of  light  scattered  from  the  sky  [60,  61].  After  this  von  Frisch’s  primary  invent 
other  researchers  began  to  investigate  polarization  vision  and  found  it  in  many  different 
species,  including  fish,  amphibians,  arthropods,  desert  ants,  and  octopuses  (see  i.e.  [63, 
64,  65,  and  66]).  These  animals  use  polarization  information  in  many  different  ways,  e.g. 
navigation,  detecting  water  surfaces,  enhancing  visibility  (similar  to  colors),  and  maybe 
even  for  mutual  communications  [67,  68,  69,  70]. 

The  pattern  of  polarization  in  an  image  of  a  scene  is  a  potentially  rich  source  of 
information.  While  the  human  eye  is  “polarization-blind”,  man-made  imaging  systems 
have  been  developed  to  collect  polarization  information  from  scene.  An  important  issue 
in  polarization  imaging  is  how  best  to  process  the  polarization  information  after  it  is 
collected  by  the  imaging  system.  In  earlier  investigations,  we  introduced  the 
“Polarization-Difference  Imaging”  processing,  inspired  by  polarization  vision  of  certain 
animal  species  [71,  57].  We  demonstrated  that  optical  imaging  systems  utilizing  PDI 
techniques  may  facilitate  the;  detection  of  .targets  in  scattering  media  even  \yhen  the. 
polarization  signal  is  rather  weak,  and  that  such  enhancement  can  increase  by  3 -fold  the 
distance  over  which  targets  can  be  detected  [71,  57,  72],  We  have  also  investigated  the 
issue  of  optimal  representation  of  the  polarization  information  to  the  “polar-blind”  human 
[73,74]. 

The  idea  of  an  adaptational  mechanism  in  polarization  vision  is  bio-inspired  as 
well.  Insects  employ  a  retinal  filter  consisting  of  an  array  of  ommatidia  (polarization 
sensitive  photoreceptors)  that  is  matched  to  the  polarization  pattern  of  the  sky  [75],  This 
filter  works  as  follows:  The  summed  output  from  all  polarization  analyzers  of  the 
polarization  sensitive  area  reaches  a  maximum  at  an  optimal  match  between  the  receptor 
array  and  the  celestial  pattern.  This  provides  the  insect  with  the  information  of  how  to 
align  its  longitudinal  body  axis  with  the  symmetry  plane  of  the  sky.  In  order  to  find  a 
proper  direction,  the  insect  has  to  change  its  orientation  and  perform  a  “check”  of 
polarization  pattern  for  the  each  angle  of  orientation. 

Using  “matched  filter”  concepts,  we  develop  here  a  polarization  imaging 
technique  based  on  the  PDI,  but  “adaptable”  to  the  environmental  conditions,  i.e.,  the 
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polarization  “background”  of  a  scene.  The  proposed  polarization-based  system  adaptively 
adjusts  its  parameters  to  obtain  the  best  possible  segregation  of  targets  from  the 
background  in  a  manner  dependant  on  the  polarization  statistics  of  the  scene. 

The  paper  is  organized  as  follows.  In  Section  2,  we  give  a  brief  overview  of 
polarization  vision  and  define  the  polarization  parameters  used  in  the  paper.  In  Section  3, 
we  describe  the  Adaptive  Polarization-Difference  Imaging  (APDI)  algorithm.  Section  4  is 
devoted  to  validation  of  the  proposed  technique,  employing  both  experiments  and 
simulations.  In  Section  4,  we  illustrate  the  performance  of  the  algorithm  in  target-against- 
background  detection  using  the  experimental  data  obtained  in  a  natural  environment. 
Finally  in  Section  5,  we  propose  an  APDI  based  method  that  may  be  used  in  surveillance 
systems. 

Physics  of  polarization  imaging 

The  polarization  of  a  local  field  of  coherent  light  source  can  be  represented  as  a 
superposition  of  two  mutually  orthogonal  wave  components.  A  phase  difference  between 
these  components  produce  linearly,  elliptically,  or  circularly  polarized  wave  with  its 
polarization  direction  determined  by  the  relative  strengths  of  the  components.  Non¬ 
coherent  light  contains  large  numbers  of  differently  polarized  light  packets.  Since  human 
eyes  and  most  normal  camera  sensors  can  only  detect  the  total  amount  of  light  energy 
collected  during  an  exposure  time,  when  the  light  energy  is  uniformly  distributed  over  all 
polarization  directions,  the  detected  signal  will  be  the  same  for  any  polarization  direction. 
Such  light  is  called  “unpolarized”  and  emitted  by  most  common  light  sources,  including 
the  Sun  or  man-made  incandescent  light  sources.  The  local  polarization  signal  at  the 
surface  of  a  sensor  (that  cannot  detect  phase  information)  can  in  general  be  described  as  a 
combination  of  unpolarized  and  completely  linearly  polarized  components. 

Assume  that  we  put  a  perfect  linear  polarizer  in  front  of  a  normal  polarization- 
insensitive  device,  such  as  a  CCD  or  film  camera.  Let  (p  represent  the  angle  that  the 

polarization  analyzer  makes  with  reference  direction,  and  the  polarization  angle  0(x,y) 
represents  the  orientation  angle  of  the  major  axis  of  the  local  polarization  ellipse  at  the 
image  pixel  position  (x,y).  Everywhere  in  this  paper,  we  use  a  right  hand  side  direction 
of  the  horizon  as.  the  reference  direction  and  angles  increase  counter-clockwise.  The 
observed  pixel  intensity  I(x,y )  can  be  described  as: 

I  {x,y,(p)  =  Iu{x,y)^L  +  p{x,y)cos\L2(d{x,y)-(p)^ , 

where  Iv  is  the  intensity  of  the  unpolarized  component,  and  p  is  the  degree  of  linear 
polarization.  The  parameter  p ,  the  “degree  of  linear  polarization”,  is  defined  as  a  ratio  of 
the  linearly  polarized  component  /  to  the  unpolarized  component  Iv :  p  varies 
between  0  and  1 ,  where  p  =  0  corresponds  to  unpolarized  light,  and  p  =  1  corresponds  to 
completely  linearly  polarized  light.  These  three  parameters,  i.e.,  Iu,  p  ,  and  6,  extracted 
for  each  pixel  (x,y)  of  the  image,  provide  a  complete  set  of  parameters  that  enable 

reconstruction  of  the  scene  image  intensity  distribution  when  view  through  at  any  given 
angle  (p  of  orientation  of  the  polarizer. 


(12) 
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At  each  pixel  of  the  scene  the  polarization  (and  intensity)  of  the  impinging  light  is 
characterized  by  three  independent  parameters:  thus,  in  order  to  provide  the  viewer  with 
complete  information  about  the  polarization  features  of  the  object,  at  least  three 
measurements  of  light  intensity  at  different  angles  q>  are  required.  These  measurements 
can  be  made  either  simultaneously  by  three  light  intensity  sensors  (e.g.,  specially 
designed  three  CCD  cameras)  when  each  sensor  has  its  polarizer  set  at  a  different  angle, 
or  sequentially  by  one  single  light  intensity  sensor  changing  the  orientation  of  the  single 
polarizer  between  the  measurements.  It  is  not  our  main  goal  to  perform  a  detailed 
comparison  of  both  methods  in  this  paper.  However,  it  is  worth  mentioning  that  our 
computational  algorithm  can  be  used  with  either  measurement  method  after  proper 
calibration. 

Consider  having  three  images  of  the  scene,  i.e.,  70,  /45  ,  /90  ,  corresponding  to 
three  angles  of  orientation  of  the  linear  polarizer,  namely,  (p  =  0“ ,  45°  and  90°  with 
respect  to  the  reference  direction,  we  can  calculate  Iv,  p  ,  and  6  at  each  image  point  as 
follows: 


Iu{x^y)=[Io{x^y)+I9o(x^y)]/2 


p(x,y)  = 


(  T  /  \V  (  ,  /  \V 


) 


+ 


l- 


tooi^y) 


f/2 


(13) 


8(x,y)  =  arctan 


Once  the  parameters  Iy,  p ,  and  6  are  computed  at  each  pixel  of  the  scene  image,  we 
can  reconstruct  the  image  intensity  for  any  angle  (p  using  Eq.  (12),  even  though  no  actual 
pictures  are  taken  with  a  polarizer  oriented  at  (p  [51]. 

The  concept  of  Adaptive  Polarization-Difference  Imaging  (APDI) 

Inspired  by  abilities  of  visual  systems  of  certain  animal  species  to  sense  polarization 
information,  in  the  earlier  works  of  our  research  group  we  introduced  [57,  71,  72]  and 
developed  [73,  74]  the  concept  of  Polarization-Difference  Imaging  (PDI).  The  proposed 
PDI  system  [71]  captures  images  of  a  scene  at  two  orthogonal  linear  polarizations.  For 
each  pixel,  located  at  (x,y),  one  gets  a  pair  of  image  intensities,  i.e.,  I0(x,y )  and 

I90(x,y)  taken  at  0°  and  90°  orientation  of  the  polarizer,  respectively.  The  “Polarization 

Sum”  (PS)  and  “Polarization  Difference”  (PD)  signals  can  then  be  computed  by  the 
following  way: 


ps/(x’y)=/9o(x’y)+Io(x>y) 

PDl(x,y)  =  /90(x,y)-/0(x,y), 


(14) 


For  an  ideal  linear  polarization  analyzer  the  PS  image  is  equivalent  to  a  conventional 
intensity  image.  Equations  (14)  can  be  more  conveniently  presented  in  matrix  form: 
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where  the  transformation  matrix 


T  = 


1 

1 


1 

-1 


has  the  eigenvectors  of  the  covariance  matrix  for  the  uniform  polarization  distribution  as 
rows  [76]. 

Using  Principal  Component  Analysis  (PCA)  [80],  Tyo  has  shown  (see  Ref.  [76]) 
that  PS  and  PD  signals  produce  the  optimum  linear  combination:  thus  the  principal 
components  (PCs)  for  a  scene  in  which  the  polarization  angle  has  a  uniform  distribution 
are  two  independent  (orthogonal)  channels,  e.g.,  I0  [x,y)  and  I90  (x,y) .  The  analysis  and 

conclusion  in  [76]  were  made  based  on  the  analogy  with  the  biology  of  color  vision  in  the 
human  visual  system  and  related  principle  component  analysis  done  by  Buchsbaum  and 
Gottschalk  [78].  In  their  scenario  the  PS  and  PD  channels  were  optimum  in  the 
information  theory  sense,  i.e.,  their  contents  were  statistically  uncorrelated,  thus  requiring 
minimum  bandwidth  [78].  However,  the  situation  may  differ  when  the  polarization  angle 
of  the  pixels  of  a  scene  has  a  non-uniform  distribution.  What  would  then  be  the  optimum 
linear  combination  for  polarization  channels  in  this  case?  Which  criterion  of  “optimality” 
is  applicable  in  this  case?  Answering  these  questions  is  the  goal  of  or  study. 

Consider  the  general  case  such  that  the  probability  density  function  of 
polarization  angle  distribution  is  arbitrary  one.  Assume  that  a  pair  of  images  each  with 
MxN  pixels  have  been  taken  with  two  different  orientations  (not  necessarily 
orthogonal)  of  the  polarization  analyzer,  i.e.,  /,  =  I  {(p  =  ) ,  and  /2  =  l{(p  =  (p2) .  We  are 

interested  in  choosing  the  “optimal”  pair  of  orientation  angles  of  the  two  channels  as  well 
as  the  “optimal”  weighting  coefficients  of  the  two  channels  to  form  a  “Polarization- 
Opponent”  image.  The  criterion  of  the  “optimality”  will  be  discussed  below.  We  call  the 
new  computed  pair  of  images  “principal  component”  (PC)  images. 

The  covariance  matrix  C  of  the  two  images  is  defined  as  follows 


£[V,M2[',1  £[VJ-.e[/, ]£[/,] 


where: 


E[W] 


1 

MN 


M  N 


m- 1  «=l 


is  the  mean  value  taken  over  the  ensemble  of  pixels  in  the  polarization  images.  Parameter 
W  is  equal  to  /,  =  I{(p-  <px) ,  I2  =  l(<p  =  <p2),  or  a  product  of  thereof.  Once  eigenvalues 

(4,^)  and  the  eigenvectors  (v,,v2)  of  the  covariance  matrix  C  (17)  are  determined,  the 


transformation  matrix  which  has  eigenvectors  as  rows  can  be  formed  as  follows: 

Ta(m)=  ~f  “  09) 

a  p 

Here  a  and  f3  are  the  components  of  the  eigenvectors.  The  signs  of  a  and  (3  may  be 
either  positive  or  negative.  The  transformation  matrix  is  formed  in  such  a  way  that  the 
first  eigenvector  corresponds  to  the  largest  eigenvalue.  With  the  “adaptive” 
transformation  matrix  (19)  thus  determined,  the  corresponding  “PC”  images  are 
computed  as  follows: 

^  £  .  (20) 

Thus  using  Equations  (20)  we  compute  a  pair  of  “adaptive”  images,  PC,  and  PC2 ,  based 
on  any  pair  of  real  images  taken  at  two  orientations  (may  be  identical)  of  the  polarization 
analyzer,  /,  and  /2 .  The  transformation  matrix  TA  is  a  function  of  the  two  polarizer 
angles  and  the  correlation  between  the  two  real  scene  images  taken  at  these  two  polarizer 
angles.  In  other  words,  TA  is  “adapted”  to  the  polarization  distribution  in  the  scene.  In  the 

previous  “non-adaptive”  PDI  [76],  [77],  the  transformation  matrix  was  fixed  (see  [76]), 
as  in  (16)  and  not  a  function  of  the  scene.  By  analogy  with  the  transformation  matrix  of 
Tyo’s  “conventional”  PDI  system,  we  consider  that  the  first  PC  (PC,)  the  “adaptive” 

analog  of  the  PS  signal,  and  the  second  PC  (PC2 )  the  “adaptive”  analog  of  the  PD  signal. 

Our  surmise  was  that  by  analyzing  the  second  PCs  of  the  given  scene,  certain  important 
features,  e.g.  presence  of  target  on  a  complex  background,  will  be  revealed. 

In  course  of  simulations  we  have  synthesized  the  images  for  all  the  combinations 
of  the  two  polarizer  orientations  where  the  angular  orientation  of  the  polarizer  is  within 
the  range  from  0  to  180  degrees  with  5  degrees  step  ( (p  and  #>  +  180°  are 
indistinguishable  cases  since  normal  cameras  cannot  detect  the  phase  information  of  light 
waves).  For  each  pair  of  “non-target”  images  corresponding  to  the  two  different 
orientations  of  polarization  analyzers  ( #>, ,  #>2 )  we  obtained  a  full  set  of  polarization 

parameters,  i.e.,  a  =  a{(px,(p2) ,  /?  =  /?(#>, ,#?2),  \  \  (#>,,#>2),  and  A,  =2,  (#>,,#>.,).  It 

should  be  noted  that  a  =  cc{(pv(p2 )  and  (3  =  /?(#>,,#>2)  always  have  the  same  shape  and 
have  theirs  maximum  and  minimum  values  for  the  same  pair  of  angles  (#>,,#>2) .  In 
addition,  maximum  value  of  a  and  minimum  value  of  being  taken  by  absolute  value 
are  equal  (the  same  is  true  for  minimum  value  of  a  and  maximum  value  of  /3),  i.e. 

KaxHAninl  “d  |«min  |  =  |/?max  |  • 

It  is  reasonable  to  assume  that  the  optimal  pair  of  angles  will  be  those  with 
corresponding  extreme  eigenvalues  or  adaptive  coefficients.  Thus,  we  have  the  following 
three  cases  to  consider,  i.e.,  the  pairs  of  polarizers’  orientations: 

1 .  The  pair  that  maximize/minimize  the  adaptive  coefficients  a  or  J3 , 
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2.  The  pair  that  maximize/minimize  the  eigenvalues, 

3.  The  pair  of  angles  which  has  a  preferential  angle  of  the  background  polarization 

as  a  bisector. 

The  last  case  is  solely  intended  for  comparison  between  the  conventional  PD  algorithm 
and  our  new  adaptive  algorithm. 

For  the  target-detection  purposes,  the  ideal  choice  of  optimal  angles  should  yield 
a  PC  image  such  that  the  variance  in  background  pixel  intensities  are  as  low  as  possible 
while  the  variances  in  the  target  pixel  intensities  are  either  at  their  original  value  or  at  a 
higher  value.  A  known  trivial  solution  to  achieve  such  condition  is  to  subtract  images 
corresponding  to  angles  obtained  by  adding  and  subtracting  45  degrees  to  the  preferential 
angle  of  the  background’s  polarization.  As  the  previous  study  indicates  [57],  selecting  a 
pair  of  angles  that  are  ±45°  from  the  angle  of  the  preferential  polarization  of  the 
background  eliminates  the  latter  and  reveals  the  target  against  it.  It  has  been  also  proven 
[76]  that  for  a  case  of  a  uniformly  polarized  background,  this  pair  of  angles  is  the  optimal 

one  with  the  corresponding  adaptive  coefficients  a  =  -J3  =  l/v2 ,  and  the  images  they 
yield  (i.e.,  PS  and  PD)  are  the  principal  components  of  the  scene.  In  the  case  of  a  non- 
uniformly  polarized  background,  however,  such  a  simple  solution  does  not  produce  the 
desired  result.  Our  extensive  simulations  on  various  computer-generated  targets  have 
shown  that  if  the  standard  deviations  of  the  polarization  parameters  are  high  enough,  the 
optimal  set  of  coefficients  is  different  from  those  corresponding  to  the  uniform  case,  and 
the  background  was  not  cancelled.  Thus,  there  is  a  need  for  a  more  general  tool  which 
can  be  used  for  scenes  with  non-uniform  polarization  distributions.  Our  study  proves  that 
more  sophisticated  and  time  consuming  calculations  than  a  simple  subtraction  (case  #3) 
are  necessary  to  detect  target  in  a  non-uniformly  polarized  environment. 

The  eigenvalues  \  and  f  as  functions  of  (px  and  (p2  are  symmetric  with  respect 

to  bisector  line  #>,  =  .  Both  the  maximum  and  the  minimum  values  of  \  are  located  on 

this  line  as  well.  This  presents  a  degenerate  case  when  both  polarizers’  angles  of  the  two 
source  images  have  the  same  orientation  and  is  obviously  not  useful.  Therefore,  case  #2 
is  reducing  to  analysis  of  behavior  of  only  smallest  eigenvalue,  i.e.  \ .  The  value  of  \ 
presents  the  variance  in  the  PC2  image- and  that  "could -imply  that  more  interesting 
information  can  be  obtained  from  the  PC2  image.  We  focused  our  study  on  cases  #1  and 
#2,  and  the  results  were  compared  with  those  obtained  for  case  #3,  which  represents  the 
conventional  PDI  technique. 

Validations  of  the  APDI  Algorithm  and  the  Selection  the  Optimal  Set  of 
Parameters 

For  the  verification  of  the  APDI  algorithm  we  conducted  several  sets  of  experiments  and 
simulations.  The  first  set  of  images  was  taking  in  our  laboratory  -  a  controlled 
environment  with  the  illumination  conditions  maintained  unchanged.  This  enabled  an 
accurate  calculation  of  sensitivity  index  in  the  context  of  target  detection.  The  target  was 
a  specially  designed  object  with  known  distinct  polarization  properties,  and  the 
background  was  kept  simple  though  still  exhibited  non-uniform  polarization  statistics. 
The  second  set  of  images  was  taken  with  real-life  targets  under  natural  (sun  light) 
illumination. 

Experimental  setup  in  the  laboratory 

The  laboratory  experimental  setup  and  a  specially  manufactured  target  are  shown  in 
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Figure  1.  Incandescent  150  W  lamp  was  illuminating  the  cylindrical  Plexiglas  tank  (12” 
height  and  16”  in  diameter)  from  a  side.  The  tank  was  filled  with  a  solution  of  a  10  mL  of 
whole  milk  diluted  in  approximately  27  L  of  water.  The  height  of  the  water  level  in  the 
tank  was  21  cm.  This  created  a  model  of  dispersive  media  (such  a  method  of  simulation 
of  scattering  media  was  originally  used  by  Tyo  [57]).  In  order  to  produce  Lambertian 
type  illumination,  an  opal  glass  diffuser  was  placed  between  the  tank  and  the  light  source. 
The  target  was  an  aluminum  disk  with  5.1  cm  in  diameter.  The  target  surface  was 
sandblasted  and  there  were  seven  1-cm2  square  patches  attached  to  it  (see  Figure  lb).  The 
six  outer  patches  were  abraded  with  emory  paper  in  such  a  way,  that  they  formed  three 
orthogonally  oriented  pairs,  i.e.,  0°  and  90°,  30°  and  120”,  60°  and  150°  (with  respect  to 
the  vertical  axis).  Patches  with  orthogonal  directions  in  the  scratches  are  located 
symmetrically  with  respect  to  the  vertical  axis.  The  surface  of  the  center  patch  was 
sandblasted  the  same  way  as  the  base  plate  surface.  The  patches  were  raised  a  few  mils 
from  the  base  plate  surface.  The  target  was  attached  to  the  Plexiglas  plate  and  facing  up. 
The  distance  between  the  surface  of  the  water  ant  the  plane  of  the  aluminum  disk  was 
55  mm.  The  target  was  observed  by  Olympus  E-10  SLR  digital  camera  with  Sunpack® 
62  mm  diameter  circular  glass  polarizer  attached  in  front  of  it.  The  images  were  taken 
sequentially  for  three  different  orientations  of  the  polarizer,  i.e.,  0° ,  45°,  and  90°  degrees 
by  rotating  the  polarizer  between  shots.  The  time  required  to  capture  all  three  images  was 
less  than  10  seconds.  The  same  Olympus  digital  camera  and  polarizer  were  used  in  our 


a)  b) 

Figure  1.  Layout  of  the  experimental  setup:  a)  Photograph  of  the  setup,  b)  7-patch 
target  together  with  a  US  10  cent,  “dime”,  coin. 


All  computer  simulations  in  this  study  have  been  done  using  “The  MathWorks 
MATLAB®”  software  package  with  its  “Image  Processing  Toolbox”.  The  captured 
images  were  stored  in  Olympus  Raw  Format  (ORF),  which  gave  us  “raw”  images  of  the 
scene,  without  any  enhancements  and  modifications  that  most  commercial  digital 
cameras  perform  internally  to  make  pictures  “look  better”.  The  Olympus  E-10  has  a 
single  chip  color  CCD  with  RGGB  Bayer  primary  color  filter.  For  our  computations,  we 
extracted  the  R,  G,  and  B  components  of  RGB  (red-green-blue)  output  directly  from  the 
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RGGB  Bayer  filter  pattern  response,  so  that  the  image  had  only  14  the  total  number  of 
active  pixels  of  the  CCD  chip  (the  G  image  is  an  average  value  of  the  two  G  filter 
responses).  This  transformation  has  been  done  by  a  specially  written  MATLAB®  code. 
The  “adaptive”  algorithm  may  be  applied  to  a  pair  of  images  representing  one  of  the 
components  (either  R,  G,  B  or  V).  The  V  (luminance)  component  of  the  HSV  (hue- 
saturation-value)  was  computed  by  the  MATLAB®  Image  Processing  Toolbox  and  was, 
in  fact,  the  maximum  value  of  the  R,  G,  or  B  channel  at  eveiy  pixel.  In  this  paper  we 
present  results  which  were  obtained  based  on  the  V  component  of  the  images. 

The  camera  zoom  was  adjusted  in  such  a  way  that  the  area  occupied  by  the 
aluminum  disk  was  only  a  portion  of  the  target  scene.  In  order  to  obtain  of  the 
“background”  scene,  we  simply  removed  the  aluminum  disk  from  the  scene  while 
keeping  all  other  experimental  conditions  including  focusing  distance  of  the  camera 
intact.  The  original  image  was  cropped  to  800  by  600  pixels  size  to  decrease  the  memory 
consumption.  The  polarization  statistics  of  the  background  scene  are  shown  in  Figure  2. 
The  histograms  of  the  polarization  parameters  reveal  an  essentially  non-uniform 
distribution  of  the  polarization  statistics  of  the  scene  with  the  average  degree  of  linear 


Figure  2.  Normalized  histograms  of  polarization  parameters  of  the  background:  A) 
Unpolarized  pixel  intensity,  7y  ;  B)  Degree  of  linear  polarization,  p ;  C)  Angle  of 

polarization,  6 . 


The  APDI  algorithm  was  applied  to  each  “target-background”  pair,  and  PCi  and  PC2 
images  for  all  three  cases  described  above  were  computed.  The  adaptive  parameters 
corresponding  to  the  cases  under  consideration  are  shown  in  Table  1 . 

Table  1  Adaptive  parameters  corresponding  to  the  cases  considered  for  the 

benchmark  target 


a 

P 

Case  #1 

145 

55 

1.116 

Case  #2 

95 

5 

0.680 

-0.733 

1.005 

3.447 

Case  #3 

100 

10 

0.707 

-0.707 

1.004 

3.327 

The  images  of  first  and  second  principal  components  for  all  three  cases  are  presented  in 
Figure  3.  Images  of  PCi  for  all  selected  cases  look  very  similar  and  non-uniform 
illumination  of  the  scene  is  clearly  visible.  Images  of  PC2  in  panels  b)  and  c )  of  Figure  3 
are  similar  because  the  pairs  of  angles  for  cases  #2  and  #3  are  close  to  each  other  as  well 
as  adaptive  parameters  (see  Table  1).  While  PC2  image  corresponding  to  case  #1  (Figure 
3a)  is  significantly  different:  fringes  of  the  disk  are  clearly  outlined  and  direction  of 
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scratches  in  all  patches  are  distinguishable.  This  is  achieved  due  to  intrinsic  properties  of 
the  APDI  algorithm,  i.e.,  decreasing  variance  in  the  background  of  the  scene.  All  images 
in  Figure  3  are  stretched  to  cover  8-bit  display.  In  order  to  better  represent  relative 
difference  in  intensity  levels  between  target  region  ant  the  background,  we  treated  each 
PC  image  as  a  3-dimentional  surface,  where  the  pixels’  coordinates  were  given  by  x  and 
y,  and  the  level  of  intensity  in  each  pixel  was  counted  along  z  axis.  The  relative 
difference  in  intensity  levels  could  be  shown  better  if  those  surfaces  were  projected  to  xz 
plane  with  the  observation  point  being  located  in  the  same  plane.  Such  surfaces  of  PCs 
for  all  three  cases  are  shown  in  Figure  4,  where  horizontal  axis  is  the  largest  dimension  of 
the  image  and  the  vertical  axis  gives  raw  values  of  pixels’  intensities.  Comparing  pictures 
in  the  right  column  in  Figure  4,  one  may  notice  that  in  panel  a )  variance  in  the 
background  of  PC2  image  is  much  smaller  (from  two  to  three  times)  than  those  in  other 
panels.  The  region  where  the  disk  is  located  is  noticeably  segregated  from  the 
surrounding  part  of  the  image  in  panel  a,  while  having  the  same  range  of  variation  in 
background  and  target  regions  for  panels  b  and  c.  Thus,  the  case  #1  may  be  a -candidate’ to" 
provide  an  “optimal”  set  of  adaptive  parameters.  Pictures  of  PC]  (left  column  in  Figure  4) 
are  very  similar  and  no  easily  noticeable  changes  between  different  cases  takes  place. 
This  is  in  agreement  with  the  images  of  PCI  shown  in  left  column  of  Figure  3. 
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Figure  3.  Principal  components  of  the  scene  corresponding  to  three  cases  of  interest. 
Left  column  show  PCi,  and  right  column  PC2  images,  respectively.  Panels  a),  b)  and 
c )  correspond  to  cases  ##  1,  2,  and  3,  respectively.  All  images  are  linearly  rescaled  to 

exploit  all  the  8-bit  displayable  range. 
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Figure  4.  Left  panels  show  PCi,  and  right -  PC2,  respectively  (shown  as  images  in 
Figure  3).  Each  image  presented  as  3-D  surface  with  pixels’  intensities  along  z  axis  is 

projected  on  xz  plane. 


Performance  evaluation:  Sensitivity  index 
A  consistent  evaluation  of  the  effectiveness  of  the  target  detection  is  important  for  this 
study.  In  addition  to  the  subjective  evaluation  of  the  resulting  images  by  human  viewers, 
we  also  employ  a  more  objective  numerical  evaluation.  In  this  section,  we  present  a 
concise  description  of  the  numerical  evaluation  method  we  use:  the  sensitivity  index 
calculation.  The  sensitivity  index  is  an  instrument  of  Signal-Detection  Theory  (SDT), 
which  is  a  measure  of  an  observer’s  ability  to  discriminate  a  target  from  a  background 
[79].  When  the  “signal-and-noise  distribution”  is  delectably  different  from  “noise-alone- 
distribution”,  the  mean  values  of  the  two  distributions  are  separated  by  a  certain  distance 
da,  called  the  sensitivity  index.  The  mean  values  are  calculated  over  a  series  of 
sequential  responses  (separated  in  time)  of  the  observer  on  the  visual  stimuli  to  be 
detected  against  the  noise.  With  the  increase  of  da ,  the  number  of  correct  decisions  is 

expected  to  increase  for  any  given  decision  process  since  the  amount  oLoverlap  between 
the  distributions  is  expected  to  increase.  In  our  study  as  the  “signal-and-noise 
distribution”  we  consider  the  scene  distribution  including  both  the  target  and  the 
background  area,  and  the  “noise-alone  distribution”  as  the  scene  distribution  without  the 
target  area. 

For  the  sensitivity  index  calculations,  we  have  done  a  sequence  of  scene  intensity 
measurements  with  temporal  separation  3  minutes  between  the  sets.  The  experimental 
setup  was  described  in  the  previous  subsection.  We  have  taken  20  sets  of  images  (each 
consisting  of  three  images  for  different  polarizer’s  orientations)  of  the  background  and  20 
sets  of  images  of  target  scenes. 

The  evaluation  of  the  sensitivity  index  was  made  on  the  arbitrary  chosen  two 
square  regions  of  5  by  5  pixels  each  located  on  the  target  surface  as  shown  in  Figure  5. 
For  each  pair  of  the  background  and  the  target  scenes,  the  average  intensities  inside  the 
referenced  regions  in  PCi  and  PC2  images  were  calculated.  As  a  result,  we  ended  up  with 
the  two  arrays  of  20  points  each  corresponding  to  the  cases  we  have  considered.  For  each 
set  of  20  points  we  have  calculated  the  mean  value  and  the  standard  deviation  and 


Page  96 


obtained  a  value  of  sensitivity  index  as: 


Mt  (£b 

j(a2T  +  a2B)/2' 


(21) 


where  jUB  and  a2B  are  the  mean  and  the  variance  of  the  background  scene,  jUT  and  a2 

are  the  mean  and  the  variance  of  the  “target”  scene  (which  contains  both  the  background 
and  the  “target”  object)  [79], 


Figure  5.  The  scheme  of  the  specially  created  target  with  regions  used  for  the 
sensitivity  index  calculation  marked  in  blue  color.  Red  lines  identify  the  direction  of 
scratches  in  the  specific  patch.  The  left  side  region  is  referred  to  as  “region  one”, 
and  the  right  side  region  is  referred  to  as  “region  two”,  respectively. 

Thus  the  values  of  the  sensitivity  index  for  PC2  images  corresponding  to  the  three 
cases  considered  were  the  following:  da  =  1.703  and  da  =  1.687,  da  =  0.772  and 

<7a  =  -1.371,  c/a  =  1.419  and  ^=-0.908,  respectively.  Here  the  first  value  of  da  in  a 

pair  corresponds  to  the  left-hand  side  region  (see  Figure  5)  and  the  second  value  to  the 
right-hand  side  region,  respectively.  Estimations  of  the  sensitivity  indices  for  different 
cases  show  that  the  APDI  algorithm  provides  the  higher  rate  of  target-against-background 
distinction  then  conventional  PDI  technique.  A  significantly  larger  value  of  sensitivity 
index  for  the  principal  component  generated  using  the  maximum  adaptive  coefficients 
gives  an  idea  that  such  coefficients  might  he  corresponding  to  the  optimal  pair  of  angles. 
The  sensitivity  of  the  APDI  algorithm  to  the  rotation  of  the  optimal  pair 
of  angles 

The  APDI  algorithm  may  be  used  as  an  effective  tool  in  a  visual  surveillance  system.  An 
important  issue  in  this  application  is:  “how  sensitive  are  the  adaptive  pairs  to  rotation  of 
the  polarization  channels?” 

Assume  that  using  the  APDI  technique  an  optimal  set  of  adaptive  parameters  was 
found.  Consider  then  the  optimum  pair  was  rotated  by  certain  angle  clockwise  or  counter¬ 
clockwise.  Would  it  be  possible  to  obtain  the  PC2  image  of  the  same  quality?  The 
simulation  on  the  sensitivity  of  the  adaptive  algorithm  use  the  same  set  of  experiment 
data  as  described  in  the  previous  section.  The  adaptive  coefficients  were  applied  to  the 
pair  of  images  corresponding  to  angles  rotated  by  5  and  10  degrees  counter-clockwise 
from  the  optimal  pair  (case  #1).  In  Figure  6  we  present  both  the  images  of  PC2  and  their 
corresponding  surface  plots  projections.  Panel  a)  is  the  same  image  as  those  in  Figure  4 a, 
panels  b )  and  c)  show  PCs  obtained  by  rotating  the  optimal  pair  of  angles  by  5  and  10 
degrees,  respectively.  Comparison  between  panels  in  Figure  6  reveals  that  by  rotating  the 
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optimal  angle  pair,  both  the  background  level  and  its  variance  are  increasing  while  the 
intensity  level  of  the  target  is  staying  on  approximately  the  same  level.  This  can  be  shown 
in  the  histograms  of  the  PC2  images  (see  Figure  7). 


Figure  6.  The  sensitivity  of  the  APDI  algorithm  to  the  rotation  of  the  optimal  pair  of 


angles  by  5  and  10  degrees  counter-clockwise.  Panels  a)  shows  same  PC2  image  as  in 


Figure  3a;  Panels  b)  and  c)  show  images  obtained  by  rotating  the  optimal  pair  by  5 
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and  10  degrees  counter-clockwise,  respectively.  Left  column  shows  PC2  images 
stretched  to  cover  8-bit  display  range.  Right  column  shows  PC2  images  as  3-D 
surfaces  projected  to  xz  plane,  where  z  axis  gives  pixels’  intensity  value. 


X10* 


Figure  7.  Histograms  of  the  PC2  images  shown  in  Figure  6.  Increasing  of  variance  in 
the  PC2  with  rotation  of  the  optimal  pair  of  angles  is  shown. 

Target  Detection  against  a  Non-Uniformly  Polarized  Background  under 
Natural  Illumination  Conditions 

In  addition  to  the  experiments  in  the  lab,  we  performed  several  experiments  under  the 
natural  illumination  conditions.  Every  experiment  session  include  capturing  three  images 
of  both  the  background  scene  and  the  target  scene.  Unlike  the  lab  experiments,  where  the 
background  is  mostly  a  uniform  scene,  here  the  background  is  a  relatively  complex  one, 
and  contains  several  different  objects. 

The  first  experiment  presented  here  was  done  inside  the  Levine  Hall  of  the 
University  of  Pennsylvania.  The  images  for  this  experiment  show  the  stone  floor  surface 
of  the  hall.  The  scene  consisted  of  the  shadow  region  formed  on  the  floor  by  the 
combined  shadows  of  an  upright  standing  person  and  the-window  frame.  The  rest  of  the 
scene  was  illuminated  by  the  sunlight  passing  through  the  window  glass.  The  camera  was 
located  opposite  to  the  window  and  elevated  above  the  floor,  thus  the  scene  was 
illuminated  by  sunlight  in  such  a  way  that  the  light  beam  coming  from  the  sun  and  a 
beam  reflected  from  the  scene  formed  the  same  plane.  The  floor  was  of  mainly  dark  grey 
color  with  a  broad  pattern.  The  “target”  object  was  a  translucent  plastic  CD  case.  We 
took  three  images  of  the  background  and  the  target  scenes.  The  camera  settings  were  kept 
the  same  for  both  the  background  only  image  sets  and  the  image  sets  with  the  target  in 
the  scene.  Figure  8  shows  three  polarization  components  (Iv,  p ,  and  0)  for  both  the 

background  and  the  target  scenes  placed  side-by-side  for  easier  comparison.  Figure  8 
shows  that  the  target  object  has  lower  degree  of  polarization  than  surrounding 
background. 

Using  our  adaptive  algorithm,  the  optimal  pair  of  angles  are  found  to  be  (px  =  140° 
and  (p2  =  50° ,  and  the  adaptive  coefficients  are  a  =  0.9965  and  J3  =  -0.0834 .  The 
histogram  of  the  background  angle  of  polarization  shows  that  the  preferential  angle  of 
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polarization  of  the  background  is  approximately  0 B  =  140°,  which  means  that  the  optimal 

angle  pair  is  the  angle  of  preferential  polarization  and  the  angle  orthogonal  to  it.  Both  the 
maximum  and  the  minimum  angles  corresponding  to  the  largest  eigenvalue  are  found  in 
the  bisector  line  of  eigenvalue  surface,  where  A^max  is  located  at  the  point  (px  =  cp2  =  0B , 

and  Af"n  is  located  at  the  point  (p^=(p2  =  dB  ±90°.  This  information  is  very  useful:  if  the 

preferential  polarization  angle  is  known  a  priori  then  the  optimal  angle  pairs  can  be  found 
directly  without  time-consuming  computation.  On  the  other  hand  this  can  be  an  effective 
way  to  recover  the  preferred  polarization  direction  in  a  scene. 

Comparing  the  principal  component  images  of  the  scene  (see  panels  b)  and  d)  in 
Figure  9)  one  may  see  a  significant  improvement  in  the  target/background  contrast  of  the 
second  principal  component  image  over  the  conventional  PD  image.  In  order  to  have  a 
complete  comparison,  we  included  also  images  that  correspond  to  the  pair  of  angles 
bringing  the  smallest  eigenvalue  to  the  maximum  (case  #2),  i.e.,  consider  images  with  the 
minimum  variance  in  intensity.  For  this  experiment,  the  maximum  value  of  ^  are  found 

at  ^,=105%  (p2  =15°  and  the  corresponding  coefficients  are  or  =  0.5016,  ft  =  -0.8650. 


Principal  component  images  created  in  this  case  are  also  presented  in  Figure  9  (panels  g) 
and  h )).  Although  an  improvement  over  the  conventional  PD  image  is  noticeable 


Figure  8.  Polarization  components  for  the  “non-target”  and  “target”  scenes  of  the 
experiments  under  natural  lighting.  The  left  panel  shows  /<,,/?,  and  9  (top  to 
bottom)  image  of  the  “non-target”  scene,  and  the  right  panel  shows  those  for  the 
“target”  scene.  The  plots  in  both  cases  are  linearly  rescaled  to  use  the  full  8-bit 

grayscale  display  range. 
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Figure  9.  Comparison  in  target  detection  between  images  obtained  by  our  new 
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adaptive  algorithm  and  by  the  conventional  PDI  algorithm.  They  are  the  principal 
components  images  obtained  from  the  images  shown  in  Figure  8.  Panels  a)  and  b) 
are  PCi  and  PC2  of  the  scene.  Panels  c)  and  d)  are  conventional  PS  and  PD  images. 
Panels  e)  and  f)  are  PCi  and  PC2  corresponding  to  the  maximum  value  of  the 
smallest  eigenvalue.  All  images  were  linearly  rescaled  to  cover  8-bit  gray  level 

display  range. 

A  few  additional  notes  should  be  added.  Since  APDI  algorithm  deals  with  the  statistics  of 
the  observing  scene,  if  the  target  occupies  only  a  small  portion  of  the  scene  adaptive 
coefficients  and  optimal  pair  of  angles  obtained  from  statistics  of  the  “background  scene” 
differs  from  those  obtained  from  the  statistics  of  “target  scene”  only  at  higher  digits  after 
coma.  Thus  APDI  algorithm  may  be  applied  to  the  target  scene  directly  without  gathering 
additional  information  from  the  background.  If  the  size  of  the  target  object  is  larger  than  a 
certain  threshold  level,  the  complete  set  of  measurements  required  to  segregate  the  target 
object  from  the  background.  Although  in  our  study  we  dealt  with  the  scenes  which  had  a 
non-uniform  distribution  in  polarization  parameters,  the  polarization  angle  of  the 
background  had  only  one  preferential  direction  of  polarization.  If  the  scene  has  two  or 
more  regions  of  significant  area  which  have  different  angle  of  preferential  polarization 
certain  improvement  may  be  required,  such  as  segmentation  of  the  scene  based  on  its 
polarization  statistics.  These  results  will  be  reported  in  our  next  paper. 

Using  the  APDI  in  Polarization  Assisted  Surveillance  System 
Based  on  the  computer  simulations  and  experiments  presented  above  we  propose  our  new 
algorithm  to  be  used  for  target-detection  purposes.  The  initial  optimal  angles  may  be 
selected  such  that  one  of  the  angles  is  equal  to  the  preferential  polarization  angle  of  the 
background  and  the  other  angle  chosen  to  be  orthogonal  to  the  preferential  angle.  In  order 
to  make  sure  that  the  selected  optimal  pair  is  the  right  one,  the  calculations  of  adaptive 
coefficients  in  a  few  points  (maybe  two  or  three)  around  the  selected  point  should  be 
performed.  The  pair  with  the  maximum  values  of  the  coefficients  are  then  used  as  the 
optical  angle  pair.  This  way  we  avoid  the  most  costly  computational  process  and 
significantly  decrease  the  computational  time  which  is  very  important  for  real-time 
applications. 

In  previous”  works  of  our  research  group  we  already  mentioned  a  polarization- 
assisted  panoramic  surveillance  system  utilizing  the  omni-directional  mirror  [81].  That 
system  processed  the  polarization  information  of  the  scene  and  used  our  novel  mapping 
techniques  [74]  for  presenting  the  information  to  the  human  observer.  The  new  APDI 
algorithm  proposed  in  this  work  will  be  combined  with  the  omni-directional  system  to 
function  as  a  powerful  polarization  based  omni-directional  surveillance  system. 

The  surveillance  system  will  operate  by  the  following  scheme.  First  the  camera 
system  will  gather  information  about  the  scene  assuming  that  all  objects  in  the 
observation  field  belong  to  the  background.  This  set  of  images,  will  be  stored  to  form  a 
database.  The  adaptive  coefficients  are  computed  based  on  the  information  stored  in  the 
database.  The  next  series  of  images,  e.g.,  20  more  images,  will  be  considered  as  the 
“target”  images  and  have  adaptive  coefficients  applied  to  them.  At  the  same  time  these 
images  will  substitute  the  initial  set  of  background  images  and  will  be  used  for 
calculating  the  next  set  of  adaptive  coefficients  if  it  is  found  that  there  is  no  target  in 
them.  This  process  will  be  repeated.  The  number  of  images  used  in  the  set  may  vary 
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depending  on  the  need  of  specific  applications  and  may  be  adjusted  by  an  operator.  Such 
a  system  will  provide  the  operator  with  an  enhanced  view  of  the  scene  and  helps  him  to 
track  the  appearance  of  the  new  “target”  objects. 

Conclusions 

We  have  developed  a  new  set  of  techniques  to  adaptively  form  an  optimum  linear 
combination  for  the  polarization  channels.  Utilizing  the  technique  of  principle  component 
analysis  for  non-uniform  distributions  of  the  polarization  state  in  an  scene,  we  can 
determine  the  optimum  linear  combination  of  polarization  channels  to  produce  the  best 
PC  images  that  distinguishes  the  target  out  of  the  background.  These  coefficients  depend 
on  the  polarization  statistics  of  the  scene,  and  can  be  adaptively  adjusted  as  the  imaging 
system  observes  different  environments.  The  optimum  combinations  for  polarization 
channels  with  unequal  weighting  coefficients  so  obtained  are  suitable  for  environments 
with  preferential  polarization  distribution  (such  as  under  water  or  in  foggy  media).  This 
new  discover)'  may  point  to  an  interesting  research  direction  in  the  polarization  vision  in 
certain  aquatic  species.  It  may  also  be  used  to  produce  images  with  higher  contrast 
between  targets  and  a  polarized  background  to  facilitate  target  detection  of  an  imaging 
system. 
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Abstract:  Shadow  is  an  inseparable  part  of  all  natural  scenes.  When  there  are  multiple 
light  sources  or  multiple  reflections  several  different  shadows  may  overlap  on  the  same 
location  and  create  complicate  patterns.  Shadows  are  potentially  good  source  of  scene 
information  if  the  shadow  regions  can  be  properly  identified  and  segmented.  However, 
shadow  region  identification  and  segmentation  is  a  difficult  task  and  shadows  not 
properly  identified  often  interfere  with  machine  vision  tasks  like  object  recognition  and 
tracking.  We  propose  here  a  new  shadow  separation  method  based  on  the  polarization  of 
light.  Polarization  information  of  the  scene  captured  by  our  polarization-sensitive  camera 
is  shown  to  separate  shadows  from  different  light  sources  effectively.  Such  shadow 
separation  is  almost  impossible  to  realize  with  traditional  intensity  only  images. 
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OCIS  Codes:  (150.0150)  Machine  vision;  (110.0100)  Image  processing;  (260.5430)  Polarization;  (230.5440) 
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Shadow  is  formed  whenever  there  is  a  light  source  and  an  occlusion  that  blocks  part  of  the  light.  With  the 
exception  of  the  ambient  light,  which  is  assumed  to  be  omni  directional,  all  light  sources  illuminate  only 
from  a  specific  direction.  Shadows  are  further  classified  into  “self’  and  “cast”  shadow.  “Self’  shadow 
refers  to  the  regions  of  an  object  that  is  not  directly  illuminated  by  a  light  source  due  to  its  surface 
orientation;  “cast”  shadow  refers  to  a  region  that  is  not  illuminated  by  a  light  source  due  to  occlusion  by 
other  objects.  Shadowed  regions  usually  appear  darker  than  the  lit  regions  and  their  color  properties  (e.g., 
hue  and  saturation)  also  appear  different  than  the  lit  regions.  Such  differences  in  intensity  and  color  create 
additional  patterns  and  boundaries/edges  that  often  confuse  human  observers  or  machine  vision  algorithms 
that  identify  and  segment  objects  using  these  cues.  For  this  reason  there  have  been  many  techniques 
developed  to  identify,  segment,  and  remove  shadows  from  an  imageor  a  video  sequence[l-19].  However, 
all  existing  works  use  only  the  intensity  and/or  color  information  of  light,  in  some  cases  also  combined 
with  temporal  and  geometric  information  available.  The  third  fundamental  property  of  light,  the 
polarization,  is  never  used  for  the  purpose  of  shadow  segmentation.  Furthermore,  many  existing  methods 
assumes  very  simple  shadow  model:  an  area  of  a  scene  is  classified  either  as  shadow  or  non-shadow.  In  fact 
it  is  possible  for  an  area  to  be  both  shadow  and  lit  area  for  different  light  sources  simultaneously,  as 
explained  below. 

Since  light  reflected  from  a  surface  can  itself  act  as  another  light  source,  in  most  real  scenes  shadow 
regions  can  overlap  and  create  quite  complicated  situations.  For  example,  a  surface  area  may  be  inside 
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some  shadow  area  for  certain  light  sources  but  not  for  other  light  sources.  There  can  also  be  “shadow 
within  shadows”,  in  which  certain  area  of  the  shadow  of  one  light  source  is  also  the  shadow  area  of  another 
light  source.  Most  existing  machine  vision  methods  tries  to  remove  shadows  altogether  and  not  much  has 
been  done  to  further  separate  complicated  overlapping  shadows.  Part  of  the  reason  is  that  this  is  an  even 
more  difficult  task  than  removing  shadows  altogether  using  traditional  vision  sensors  that  detect  only 
intensity  and  color  information  of  light. 

Polarization  is  an  integral  property  of  light,  but  unaided  human  eyes  and  most  machine  vision  cameras 
cannot  detect  this  property.  However  it  has  been  found  that  many  animal  species  can  “see”  the  polarization 
information  and  make  good  use  of  it  to  their  advantages  [20-23].  Inspired  by  biological  polarization  vision, 
our  group  has  used  polarization  sensitive  cameras,  to  enhance  target  delectability  in  scattering  media  and 
for  enhanced  surveillance  [24-26].  We  have  also  developed  methods  to  display  these  polarization 
information  to  human  observers[27,28].  Although  one  of  the  primary  sources  of  light,  the  sun  light,  is  not 
polarized  in  the  visible  band  before  entering  the  atmosphere  of  earth,  the  light  reflected/scattered  from 
object  surface  is  often  at  least  partially  polarized.  In  this  work  we  show  that  complex  overlapping  cast 
shadows  that  are  almost  impossible  to  distinguish  in  the  intensity  and  color  only  images  can  be  easily 
segmented  from  each  other  in  polarization  parameter  images. 


2.  Relations  between  Polarization  Parameters  and  Shadows 


(A) 


(B) 


Unpolarized 
incident  light 


Specular  reflection 
Diffuse  reflection 


Highly/fully  polarized 
specularly 
reflected  light 


Fig.  29  (A)  General  macroscopic  reflection  model  (B)  Polarization  of  light  resulting  from  specular  reflection 
from  a  diclcctricrsirrfacc.  -  ...  - 


Light  is  an  electro-magnetic  wave  which  is  by  nature  a  transverse  wave.  This  means  that  the  electric  field 
vector  E  of  light  oscillates  along  a  direction  perpendicular  to  the  direction  of  travel  of  the  light.  This  is  the 
direction  of  polarization  of  light.  Since  most  natural  light  is  not  coherent,  when  there  is  equal  distribution  of 
light  energy  polarized  in  all  direction,  the  observed  light  intensities  does  not  show  any  preference  in 
polarization  direction  and  the  light  is  considered  unpolarized.  However,  when  light  is  scattered  or 
reflected/refracted  there  will  be  selective  filtering  of  light  energy  according  to  their  polarization  so  the 
resulting  light  will  have  a  preferential  direction  of  polarization.  See  Fig.  29  (A),  the  generally  accepted 
macroscopic  reflection  model  is  that  light  incident  on  a  surface  will  be  partially  reflected  specularly  and 
partly  reflected  diffusely.  The  ratio  of  energy  reflected  diffusely  and  specular  is  surface  dependent  and  also 
relates  to  the  angle  of  incidence.  The  diffusely  reflected  components  often  considered  undergone  multiple 
random  reflection  microscopically  so  statistically  can  be  considered  unpolarized.  However,  the  specular 
reflection  component  is  usually  at  least  partially  polarized.  This  can  be  explained  using  the  Fresnel 
reflection  coefficients  formula[29]: 
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”i«  cos° C0S°„  _  sin(Q,„  -<S>„) 
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(22) 

”,r  C0S(I)ln  ~nin  C0S<^„  _  tanC^fl,  -^) 

,,  «mcos(D,r  +  «,rcosOm  tan(<D,n+0,r) 

where  subscripts  ‘in',  ‘tr’,  and  ‘re’  stand  for  incident,  transmitted,  and  reflected  component;  r  the 
reflectance;  E  stands  for  the  amplitude  of  the  electric  field;  n  the  index  of  refraction  of  the  media.  .  The 
subscripts  ‘  JL  >  and  ‘  ||  ’  are  related  to  the  plane  of  incidence.  In  Fig.  29(B)  the  plane  of  this  paper  is  the 
plane  of  incidence.  In  case  of  the  specular  reflection  it  contains  both  the  incident  and  reflected  light  wave 
vectors.  It  is  clear  that  the  reflectance  of  light  that  is  parallel  polarized  and  perpendicularly  polarized  is  in 
general  not  the  same.  In  particular,  can  be  exactly  zero  at  the  Brewster’s  angle  0B  ,  which  is  given  as: 

tan  0B=— .  :  (23) 

ni 

In  this  case  the  reflected  light  will  be  completely  polarized  along  the  direction  perpendicular  to  the  plane  of 
incidence.  In  the  visible  band  practically  all  observed  light  from  most  object  surface  is  reflected  light.  If  we 
compare  the  light  observed  in  a  cast  shadow  area  to  the  light  observed  from  area  lit  by  the  light  source,  it  is 
clear  from  Eq.  (22)  that  they  will  in  general  have  different  polarization  properties. 

Another  important  source  of  polarized  light  is  the  scattering  of  light  by  the  atmosphere  of  the  earth. 
The  scattering  of  sun  light  by  the  air  particles  can  be  explained  by  the  theory  of  Rayleigh  scattering[ 29], 
which  models  the  particles  as  an  electric  dipole  and  the  scattered  light  as  results  of  dipole  radiation.  Since 
the  dipole  radiation  cannot  radiate  energy  at  directions  along  the  direction  of  oscillation,  the  scattered  light 
observed  at  a  right  angle  from  the  incident  light  direction  will  be  polarized  parallel  to  the  plane  that  is 
orthogonal  to  the  direction  of  incident  light.  The  scattering  is  also  wavelength  dependent: 

P°c(l//t)4  (24) 

where  P  stands  for  the  radiance  power  and  2  the  wavelength  of  light.  This  is  the  reason  we  see  a  blue  sky  in 
the  sunny  day.  What  we  do  not  perceive  with  our  unaided  human  eyes  is  that  the  sky  light  is  also  polarized, 
the  more  so  from  directions  90  degrees  from  the  position  of  the  Sun.  We  show  in  this  work  that  such 
natural  phenomenon  can  be  used  to  neatly  separate  complex  shadows.  For  example,  an  area  that  is  inside 
the  shadow  caused  by  the  direct  sun  light  but  is  lit  by  the  polarized  ambient  sky  light  will  show  certain 
polarization,  while  the  are  that  is  in  both  the  shadow  area  of  the  direct  sun  light  and  the  sky  light  will  show 
no  polarization  at  all. 

In  addition  to  the  linearly  polarized  case,  there  exist  cases  called  circular  and  elliptical  polarization  that 
can  result  from  two  coherent  but  orthogonally  polarized  light  components  in  the  same  frequency  but  with 
different  phase  lags[29].  However  most  imaging  sensor  can  only  sense  the  integration  of  light  energy  over 
time  and  the  phase  information  is  not  recorded.  The  general  expression  for  observed  intensity  of  partially 
polarized  light  I  as  a  function  of  the  angle  of  orientation  of  polarization  analyzer  <p  can  be  written  as 
follows: 

l(<P)  =  lu  +/<cos[2(0-^)]  =  /y{l  +  /?cos[2(0-^)]}  (25) 

where  6  is  the  orientation  angle  of  the  major  axis  of  the  polarization  ellipse,  lv  is  a  half  of  the  total  pixel 
intensity,  and  p  =  IAHu  is  the  degree  of  linear  polarization  at  the  pixel.  The  reference  axis  for  <p  and  8 

can  be  arbitrarily  chosen.  Since  the  exact  index  of  refraction  of  the  surface  in  our  study  is  considered 
unknown,  putting  one  polarizer  at  a  given  orientation  angle  in  front  of  the  camera  and  taking  a  picture  can 
not  provide  the  complete  information  about  the  polarization  state  of  the  received  light.  By  taking  three 
pictures  with  the  polarizer  oriented  at  three  different  angles,  for  example  <p  =0,  45  and  90  degrees,  we  can 
recover  Iu,IA  ,  and  6  for  each  pixel  of  the  image  using  the  following  expressions: 
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>u=(> o+4o)/2 

/,  =  \/a.s-4)!+(/»-4)! 

0  =  arctan  [(/„  -  /„  )/(/„-/„  )]/2 

Here  indices  0,  45,  and  90  indicate  the  orientation  of  the  polarizer  in  degrees  when  the  image  was  taken. 
Because  6  and  6+n  are  indistinguishable  for  phase-blind  visual  sensors  in  most  conventional  cameras,  the 
meaningful  range  of  6  is  restricted  by  n.  We  usually  use  6  range  from  0  to  n.  A  system  of  two  cameras  with 
a  controllable  polarizer  and  a  beam  splitter  to  rapidly  take  the  required  pictures  has  been  developed  in  [30]. 

There  are  other  natural  processes  that  can  produce  polarized  light,  like  dichroism  (polarization 
produced  via  selective  absorption),  birefringence  (different  index  of  refraction  for  differently  polarized 
light),  ...etc.  are  less  common  in  natural  environment  will  not  be  considered  here. 

3.  Experiments 

The  first  example  is  an  outdoor  scene  of  a  walkway  in  front  of  a  building  with  all-glass  walls.  See  Fig.  30, 
the  Sun  is  illuminating  from  the  right  hand  side  of  the  picture  with  shadows  cast  by  trees  along  the 
walkway.  Most  existing  shadow  handling  algorithm  will  most  likely  just  segment  out  the  dark  areas  as 
shadow  areas  or  remove/reduce  the  contrast  in  brightness  caused  by  the  shadow.  However  there  is  more 
complicated  overlapping  shadow  patterns  hidden  inside  the  scene  that  are  not  detectable  using  only 
intensity,  color,  and  even  temporal  and  geometrical  information  used  by  existing  shadow  handling 
methods. 


Fig.  30  Left:  ordinary  intensity  image  of  an  outdoor  scene  with  light  and  shadow.  Right:  the  same  scene 
showing  the  degree  of  polarization  image.  Hidden  patterns  of  shadows  within  shadows  are  clearly  visible  in 
high  contrast.  ..  ■  ...... 

In  this  scene  the  glass-wall  building  to  the  left  hand  side  of  the  picture  is  reflecting  the  sun  light  from 
the  glass  part  but  not  from  the  thin  frame  part  around  each  piece  of  glass.  The  reflected  light  is  polarized 
and  the  reflection  pattern  is  cast  on  the  scene  overlapping  with  the  easily  visible  shadow  pattern  caused  by 
the  direct  sun  light.  The  reflected  light  is  weaker  than  the  direct  sun  light  and  the  pattern  is  essentially  not 
visible  in  the  intensity  only  image.  However,  when  we  use  polarization  sensitive  camera  and  extract  the 
degree  of  polarization  image  these  hidden  pattern  of  overlapping  shadow  is  revealed,  see  Fig.  30  Right.  The 
area  that  is  neither  lit  by  direct  sun  light  nor  by  the  reflected  light  from  the  glass  is  both  dark  and 
unpolarized.  These  are  the  cast  pattern  of  the  glass  frames  of  the  glass-wall  building  to  the  left  of  the 
picture.  The  area  that  is  not  lit  by  the  direct  sun  light,  thus  appear  as  part  of  the  shadow  area  in  the  intensity 
image,  but  is  lit  by  the  polarized  reflected  light  from  the  glass  pieces  of  the  glass-wall  building,  exhibits 
strong  polarization.  The  degree  of  polarization  image  normalize  the  polarization  effects  with  respect  to  the 
total  intensity  as  shown  in  Eq.  (26),  so  these  area  will  show  up  as  bright  area  in  the  degree  of  polarization 
image  in  Fig.  30  Right.  To  show  that  these  glass  piece  patterns  revealed  by  the  degree  of  polarization  image 
is  unique  and  not  hidden  in  the  intensity  only  images  due  to  poor  contrast  in  the  shadow  area,  we  conduct 
linear  contrast  enhancement  and  followed  by  gamma  correction  of  0.5  to  both  images  shown  in  Fig.  30  and 
the  results  are  shown  in  Fig.  31.  It  is  clear  that  the  hidden  shadow  patterns  are  only  revealed  in  the  degree 
of  polarization  image.  Fig.  32  Left  shows  the  glass-wall  and  frames  of  the  building.  Fig.  32  Right  is  the 
same  walk  way  when  the  bright  direct  sun  light  is  blocked.  This  shows  that  the  patterns  revealed  in  the 
degree  of  polarization  image  is  indeed  caused  by  the  hidden  shadow. 
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Fig.  3 1  Images  in  Fig.  30  contrast  enhanced  by  linear  intensity  range  stretch  followed  by  gamma  correction  of 
0.5  to  show  the  details  in  the  dark  area.  Left:  intensity  image.  Right:  degree  of  polarization  image.  It  is  clear  that 
the  pattern  revealed  in  the  polarization  image  is  not  present  in  the  intensity  image  even  after  contrast 
enhancement. 


Fig.  32  Left:  the  glass-wall  building  showing  big  glass  rectangles  and  frames.  Right:  A  picture  of  the  same  walk 
way  as  in  Fig.  30  and  Fig.  3 1  taken  another  day  when  the  direct  sun  light  is  blocked  due  to  nearby  construction 
scaffolding.  The  shadow  pattern  cast  on  the  walk  way  by  the  glass-wall  and  frames  is  visible. 

We  performed  a  lab  controlled  experiment  to  fiirther  confirm  our  results  obtained  outdoors.  The  setup 
is  a  150W  incandescent  light  illuminating  from  the  opposite  side  of  the  camera  and  a  15W  fluorescent  light 
illuminating  from  the  right  hand  side  of  the  picture. 


Fig.  33  A  metal  pillar  on  a  optical  table  illuminated  by  a  strong  incandescent  light  from  the  side  opposite  to  the 
camera,  while  another  much  weaker  fluorescent  light  illuminating  from  the  right  hand  side  of  the  picture.  Left: 

Intensity  only  image.  Right:  Degree  of  Polarization  image. 

See  Fig.  33.  In  the  intensity  only  image  only  the  shadow  of  the  dominating  light  source  is  visible. 
However,  in  the  degree  of  polarization  image,  both  shadows  are  visible  and  separated  clearly  in  high 
contrast.  The  “shadow”  of  the  much  weaker  light  from  the  right  hand  side  shows  up  as  a  bright  area  to  the 
left  of  the  metal  pillar.  The  reason  is  that  the  strong  light  reflected  from  the  table  is  polarized  because  the 
view  geometry  while  the  side  illuminating  light  is  very  weakly  polarized  due  to  the  view  geometry.  As  a 
result,  the  area  that  is  not  lit  by  both  light  source  is  totally  dark,  and  is  least  polarized  and  show  up  as  the 
darkest  area  in  the  degree  of  polarization  image.  The  area  that  is  lit  by  both  the  strong  light  source  and  the 
weak  light  source  is  polarized  but  the  degree  of  polarization  is  weakened  by  the  unpolarized  irradiance 
caused  by  the  weak  unpolarized  light  from  the  right  hand  side  of  the  picture.  The  shadow  area  of  the  weak 
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light  source  is  almost  invisible  in  the  intensity  only  image.  However  such  area  is  the  most  highly  polarized 
area  because  it  is  polarized  due  to  reflecting  the  strong  light  source  and  its  degree  of  polarization  is  not 
reduced  by  the  unpolarized  reflection  of  the  weak  light  source  from  the  right  hand  side.  The  contrast 
between  the  revealed  shadow  due  to  weak  side  illumination  and  the  neighboring  area  is  very  good  so  it  is 
very  suitable  for  many  well  known  segmentation  method  to  extract  this  area.  Fig.  34  left  shows  one 
example  segmentation  using  region  growing  that  starts  with  2  by  2  regions.  The  side  shadow  area  is  cleanly 
separated  from  the  image  when  21  or  more  regions  are  segmented.  Fig.  34  shows  the  result  of  extracting 
the  side  shadow  area  from  the  degree  of  polarization  image. 


Fig.  34  Left:  Segmentation  results  from  region  growing  of  Fig.  33  Right  into  21  regions.  Right:  Flidden  shadow 
area  extracted  from  Fig.  33  Right. 

4.  Discussions 

All  existing  shadow  handling  methods  have  limitations.  Many  methods  are  designed  to  handle  specific 
applications  like  aerial  photography  or  traffic  monitoring,  so  that  the  lighting  condition  is  simplified  or  can 
be  known  a  priori.  There  may  be  requirements  of  known  geometry  of  the  scene  or  known  precise  pixel 
sensitivity  calibration  data  of  the  camera.  The  polarization  based  shadow  segmentation  method  suggested 
here  also  has  limitations.  While  this  method  is  not  strictly  tied  to  a  specific  application,  it  does  require  that 
not  all  light  sources  come  from  the  back  the  camera  when  all  light  sources  are  not  polarized.  This  is 
because  that  Eq.  (22)  works  best  with  specular  reflection  which  is  seen  most  strongly  when  the  light  is 
reflected  in  front  and  toward  the  viewing  camera.  There  are  several  methods  available  that  can  estimate  the 
illumination  directions  of  multiple  light  sources  using  information  in  the  image[31-34].  Combining  our 
polarization  detection  methods  with  such  illumination  direction  estimation  methods  will  be  very  promising 
and  is  our  work  in  progress.  This  polarization  shadow  detection  method  will  also  be  very  useful  when 
combined  with  other  well  known  shadow  cues  like  intensity,  color,  and  geometry  to  achieve  more  accurate 
shadow  segmentation  and  give  more  detailed  information  on  the  origin  of  each  shadow  components.  As 
demonstrated  in  our  experiments,  the  information  found  by  polarization  is  unique  is  cannot  be  found  by 
other  shadow  cues  alone. 

5.  Conclusion 

We  have  presented  a  novel  shadow  segmentation  method  using  the  natural  polarization  property  of  light. 
The  polarization  of  light  conveys  important  information  about  the  scene.  Polarization  have  been  used  in 
many  other  vision  tasks  like  removing  glare  and  removing  glare  but  have  not  been  used  to  aid  the 
segmentation  of  complex  shadows  in  a  scene.  Many  existing  shadow  handling  methods  not  using 
polarization  can  only  handle  simple  shadow  geometry  caused  by  one  light  source.  Our  use  of  polarization 
enable  us  to  deal  with  the  complex  situation  of  multiple  light  sources  causing  multiple  overlapping  shadows 
that  is  difficult  to  even  detect  in  the  traditional  intensity  only  images.  The  proposed  polarization  shadow 
segmentation  method  also  provides  a  new  shadow  verification  criteria  that  can  potentially  be  integrated  into 
a  shadow  analysis  algorithm  together  with  other  well  known  shadow  cues  to  provide  more  accurate  shadow 
segmentation. 
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